Stefan Novakovic

Calling all #statisticians, #programmers, #dataanalysts and #datascientists - you don't want to miss this conference! Explore new horizons with Clivoffective at The Effective Statistician conference in fall 2024. Dr. Alexander Schacht, Evelyn Hund and André Hund as well as many other speakers will provide you with valuable tools for your daily business through presentations, inputs and workshops. Take a look at the program now and be sure to register today. You can find the link in the comments👇🏼

26

2 Kommentare

Data Scientists, This is BIG! Snowflake's Arctic LLM is Here – and Open-Source! 🤯 Get your hands on it now: Interactive Demo: Explore Arctic's capabilities with this Streamlit app → https://lnkd.in/gJ7y38Ng Replicate: Ready to experiment? Find it on Replicate → https://lnkd.in/gyDXiMju Why This Matters: Arctic lets you quickly build demos of your models. This is a game-changer for showcasing your work and making your models more accessible! Huge thanks to Snowflake for making this happen! Are you excited to try Arctic? Let me know in the comments! 👇 #LLM #OpenSource #DataScience #Streamlit #Snowflake

8

in Pandas 2.2 one can set pyarrow as the default backend. in this presentation the advantages of arrow vs numpy are presented: 1. Faster string operations 2. Consistent missing value handling 3. Efficient data serialization Minute 24 https://lnkd.in/gbbGpR56

22

Till today, even two years after transitioning from a non-tech background into data analytics: 👉🏾 I am still afraid to share "insights" from my analyses. 👉🏾 I am still afraid to take new steps because I don't feel ready yet 👉🏾 I am still afraid to take constructive criticism on my work. 👉🏾 I am still afraid to ask clarifying questions during data analysis tasks. 👉🏾 I am still afraid to admit when I miss patterns in the data. 👉🏾 I am still afraid to seek help when most needed. 👉🏾 I am still afraid that I don't have all the answers when asked "data questions." Nonetheless, I face these fears WHILE afraid weekly, if not daily! Whether these fears eventually die down or not, what matters is that we are not letting them hold us back from our potential! #dataanalytics #businessintelligence #data

193

21 Kommentare

All models are wrong, but some are useful. I recently stumbled upon a discussion about the well-known log-log models, which are often hailed as the industry standard for estimating elasticities—especially in retail, where products are homogenous. But in my humble (and statistically significant 😊) opinion, the best way to estimate elasticities is through causal inference followed by experiments. See: https://lnkd.in/e8eFHqWc I'd love to hear from fellow practitioners. What methods do you use —particularly for heterogeneous products like fashion items (where one size never fits all) or, in my current adventure, short-term vacation rentals?

9

Trino is a popular query engine in data lakehouses. But Trino does not support HA because trino coordinator has single point failure. There can be also bottleneck of trino query executions in monolithic trino cluster. Such monolithic approach with single large trino cluster can lead to be problematic. We need to separate a large trino cluster to small trino clusters for the groups like BI team, data scientist team, and data engineer team individually. Chango(https://lnkd.in/gsMam-hz) provides Chango Trino Gateway(https://lnkd.in/gBV7ktDw) with the following features. - Strong RBAC data access control with catalog, schema, table level for trino users. - Route trino queries to the backends trino clusters intelligently and dynamically. - Easy way to activate and deactivate trino clusters in order to update trino configurations. - Zero downtime for trino query executions. Let's see an example in Chango. You want to ingest files data or streaming events to iceberg tables and explore and transform iceberg tables with trino. 1. Load files to iceberg tables. Load files like CSV, JSON, Parquet, ORC to iceberg tables in Chango using SQL simply. 2. Ingest streaming events to iceberg tables. Just send streaming events through REST to ingest streaming events to iceberg tables in Chango without the need of streaming platform and streaming jobs. The small data files created everytime streaming events are ingested to iceberg tables will be compacted, and iceberg table maintenance like snapshot expiration and old metadata files removal will be done for you automatically. 3. Explore and transform iceberg tables. Behind Chango trino gateway, there can be several trino clusters which connect to Chango REST Catalog. Chango REST Catalog is iceberg REST Catalog. All the iceberg supported engines like trino and spark can connect Chango REST Catalog seamlessly. Chango REST Catalog provides automatic iceberg table maintenance like data file compaction, snapshot expiration and old metadata removal for your thousands of iceberg tables. With Chango Trino Gateway, trino users can explore and transform tables with strong privileges. For example, users of Data Engineer team can write the tables of schema A, but users of BI team cannot write the tables of schema A.

5

Many forecasting use cases lack the data volumes to benefit from deep learning. The rise of time series foundation models (TSFMs) is changing this landscape. Here's a summary of the latest approaches and their potential impact. 𝐓𝐡𝐞 𝐃𝐚𝐭𝐚 𝐒𝐢𝐳𝐞 𝐂𝐡𝐚𝐥𝐥𝐞𝐧𝐠𝐞 Companies like Amazon, Google, and Zalando rely heavily on deep learning models for forecasting. Yet, practitioners are often disappointed when they try deep learning on their use cases. Use cases with < 10,000 time series never benefit from deep learning scaling laws. This situation mirrors natural language processing (NLP). Typical NLP use cases can't train effective transformer models because of limited document availability. Instead, they use foundation models trained on massive datasets by selected companies. These models, having learned the general structure of language, can be adapted to specific tasks through few-shot prompting, retrieval-augmented generation (RAG), or fine-tuning. 𝐈𝐧𝐭𝐫𝐨𝐝𝐮𝐜𝐢𝐧𝐠 𝐓𝐢𝐦𝐞 𝐒𝐞𝐫𝐢𝐞𝐬 𝐅𝐨𝐮𝐧𝐝𝐚𝐭𝐢𝐨𝐧 𝐌𝐨𝐝𝐞𝐥𝐬 (𝐓𝐒𝐅𝐌𝐬) TSFMs follow a similar concept. Companies with extensive compute and data resources develop foundation models with robust generalization capabilities. These models can then be customized for specific use cases. This means benefits of TSFMs are: 1) Even small-scale use cases can benefit from advanced deep learning methods, with the prospect of using multi-modality in the future (see below; comments) 2) TSFMs can deliver impressive accuracy even in scenarios where standard methods (like XGBoost) struggle. An example are cold-start problems. 𝐓𝐲𝐩𝐞𝐬 𝐨𝐟 𝐓𝐒𝐅𝐌𝐬 There are two primary approaches to TSFMs: 1. Pre-trained Models from Scratch: Built on vast sets of curated time series data. Examples: TimesFM (Google), TimeGPT (Nixla), ForecastPFN, and LagLlama. 2. Bootstrapped from LLMs: Use the hidden structure in sentences that LLMs are trained on, viewing them as time series. Examples: Chronos (Amazon) and TimeLLM. The choice of approach depends on the specific use case, including the methods for tailoring and explaining the models. 𝐋𝐨𝐨𝐤𝐢𝐧𝐠 𝐀𝐡𝐞𝐚𝐝 TSFMs hold promise for widespread deep learning adoption in forecasting. Many challenges remain, e.g. the integration of custom covariates, but these will be addressed over time. Meanwhile, the opportunity is vast, with forecasting being crucial across industries and the options to improve this models further is huge too. For example, TSFMs could become multi-modal. They could, for example, integrate news articles for more comprehensive demand forecasting. I'm excited to see TSFMs grow and revolutionize forecasting, making advanced deep learning accessible and effective across a broad range of applications. Check out the overview article in the comments (image from there) and additional resource! #forecasting #deeplearning #machinelearning

173

10 Kommentare

I wrote the same Snowflake UDF in both Python and Java. Performance was not the main driver, but I was interested to see how the two UDFs compared. Here is what I found: The query running the UDF ingests about 16 million records and outputs 13 million records. The UDFs parse raw text into an OBJECT. I plan to use them in a dynamic table pipeline. The query with the Java UDF ran in 14.5 minutes and the average UDF execution time was 0.16ms per call. The query with the Python UDF ran in 25.5 minutes and the average UDF execution time was 0.49ms per call. The Java UDF was 3 times faster. I expected Java to be faster, but wasn't sure how much to expect. The last step of the Java UDF is to serialize the output into a JSON string. This takes up almost 1/3 of the execution time. This overhead is not present in Python, where I can return a raw dict back to Snowflake. I expect Java UDFs that return value types to execute even faster, compared to Python. The dev experience with Java was not as nice. Dealing with sql.DateTime values is rather irritating when you have timestamps with timezones. I might look into receiving the timestamp values as strings and parsing into a better class than sql.DateTime Other than timestamps, the Java experience was fine. I wish Snowflake used a different type of class for timestamps that allow us to directly set the timezone (or work exclusively in UTC). My takeaway is: writing UDFs in Java can be worth it for the performance improvement, especially when you are in startup mode and trying to minimize costs before go-to-market. I had another primary motivator, but the improved performance is quite compelling.

22

2 Kommentare

That's awesome! Mistral's new features for building custom LLMs sound incredibly developer-friendly. The ability to fine-tune and create custom agents makes it a powerful tool for personalized AI solutions. Thanks for sharing the video link—can't wait to check it out! 🚀🔍 #CustomLLMs #AIInnovation #DeveloperTools #MachineLearning #MistralLLM

1

LLMs do approximate retrieval of stored information on a computer, and that's great for many tasks, but not directly comparable with human smarts

4

1 Kommentar

Data ingestion to a Lakehouse. There are a numerous way to ingest data from a variety of sources to open #lakehouse platforms like Apache Hudi - Spark, Flink jobs. However, Hudi's platform also natively offers a utility called "Hudi Streamer" for efficient data ingestion. 𝐇𝐞𝐫𝐞 𝐚𝐫𝐞 5 𝐭𝐡𝐢𝐧𝐠𝐬 𝐭𝐨 𝐤𝐧𝐨𝐰 𝐚𝐛𝐨𝐮𝐭 𝐇𝐮𝐝𝐢 𝐒𝐭𝐫𝐞𝐚𝐦𝐞𝐫: ✅ It enables you to ingest data from sources such as distributed file system, Kafka, Pulsar, JDBC, etc. ✅ Hudi Streamer is part of the 'Platform services' that sits atop table services (such as clustering, file sizing) interfacing with writers & readers. ✅ It support functionalities like automatic checkpoint management, integration with schema registries such as Confluent, and deduplication of data. ✅ There are also features for backfills, one-off runs, & continuous mode operation with Spark/Flink streaming writers. ✅ Allows creating both Copy-on-Write (CoW) & Merge-on-Read (MoR) tables These type of platform services really makes Hudi unique from other metadata table formats. Hudi Streamer is also one of the widely used utility that comes natively with the 'table format' along with other transactional capabilities. Linked a great blog (in comments) from Amazon Web Services (AWS) on how to use Hudi Streamer with AWS Glue to ingest streaming data from Amazon MSK. #dataengineering #softwareengineering

66

2 Kommentare

It can feel tedious and boring to pretend and DEFEND against NULLs in all the data we deal with. it almost feels overkil. Here are top ways to Handle Nulls in Database. 1. IS NULL / IS NOT NULL: Use when you absolutely need to know if data is missing. WHERE critical_info IS NOT NULL 2. IFNULL: consider if a default makes sense. IFNULL(user_preference, 'system_default') 3. COALESCE: It returns the first non-NULL value in a list. COALESCE(primary_contact, secondary_contact, 'Unknown') Follow me Sampath Vijayan for more data engineering insights ! #DataEngineering #SQLTips #Database #Mysql #Postgres #Oracle #Sqlserver #Db2

9

working with DAX has never been so convenient 💛 I really like the DAX query view for mass formatting (here you also can watch the great tutorials of Marcus Wegener and Lars Schreiber) - and also mass editing of measures The use case in my tutorial is how you can mass edit measures when they are broken after I changed a Calendar table. 🤓 I also used the bulk edit feature for efficient migration of a report to Direct Lake in a scenario when I was not allowed to use any external tools or the sempy-labs. Further I think that this is a game changer for Mac Users to work with DAX like Steffen Weiler 😉 ▶ watch now in the BI or DIE blog (link in the comment) #powerbi #dax

12

2 Kommentare

Andreas Scherbaum: PostgreSQL Development Conference 2024 - Review 🦸‍♂️[PostgreSQL Geeks Unite]🦸‍♀️ 🚀 Press pause on your index optimization and gather 'round! Breaking news from the PostgreSQL universe: PGCon is now PGConf.dev. Yes, you read that right. And no, you won't be catching that plane to Ottawa; get your passports ready for Vancouver, BC! 🎤 This is not a drill, my SQL wizards! New city, new vibe, new team, who absolutely nailed it this year. Imagine all your favorite extensions and modules compiled into one seamless conference experience. Shoutout to Andreas Scherbaum for the fantastic review. You had me at "PGConf.dev." Clever name, guys. Really rolls off the tongue, doesn't it? Whether you're replicating the good times or just trying to avoid deadlocks in your schedule, PGConf.dev is clearly the place to be. Pro tip: Book your flights, brush up on your CTEs, and consider this an official hint to present next year (yes, you, with the million row query and the trick up your sleeve). Let's keep those transactions committed and our spirits high. Until next time, keep your queries optimized and your connections pooled. #PostgreSQL #PGConfDev #DatabasesAreFun #TravelForTech 🔍👀 🧠 Credit: Andreas Scherbaum You can find the full article here: https://postgr.es/p/6x6

1