Sumit Rane

As models grow larger with the rise of LLMs, choosing the right precision type has become important. Sharing some insights over FP16 vs BF16:   🔷  FP16 and BF16 both consume same memory.  🔷  FP16 has better precision but lower dynamic range. In fp16, largest number is 65,504. BF16 can go upto 3.39e+38 !  This is because fp16 has higher # mantissa bits and bf16 has higher #exponent bits. 🔷  For model training, you will have to experiment and understand whether your network is sensitive to range or precision. If a model requires more dynamic range, it will converge better with bf16, which has same range as fp32. However, some models will converge better with higher precision (fp16) 🔷  When finetuning an fp16 model pre-trained in bf16, overflow issues may arise as some of the bf16 model weights might be larger than 65k.   Image source: nvidia   #precisiontype #fp16 #bf16 #modeltraining #LLMs #machinelearning #ai #datascience #torch

30

❝ How do you taste your Wine? With the right Glass, served from the right Bottle. How do you utilize your LLM? With the right Task, served from the right Compute environment. ❞ If the quote above caught your interest, check out my blog post that was featured on the Toyota Connected India Medium channel last week: Leveraging LLMs in Cloud - 🔗 https://lnkd.in/gfHsdevH --- I hope this blog adds a bit of amusement for the seasoned AI/ML engineers 🧠👨‍💻 with the carefully crafted images and perhaps even the silly wine analogies 😉. Whether you're a cloud engineer ☁️ 👨‍💻, a product owner of cloud-based apps, or someone new to Generative AI in the cloud, this blog offers a detailed, beginner-friendly look into the selection and deployment of LLMs in cloud environments. --- A heartfelt thanks 🙏 to my colleagues at Toyota Connected India for taking the time to review, shape this piece and even share a creative writeup like below in the original post 🫡 👇 🤔 Curious to dive in? Click the link below to explore the blog on Medium! 🌐 🔗 https://lnkd.in/gfHsdevH

18

Many of ML practitioners might have heard the term "Extending context length" for LLMs. In case if you want to dive deeper into this topic along with its code implementation, give this blog a read. I have described about Position Interpolation, NTK aware and Dynamic NTK method, which are widely used for extending context length.

68

Another important point to consider when interpreting PR AUC values is that they are typically lower than ROC AUC values. This difference becomes clear when we understand the baseline metrics for each curve: Baseline ROC AUC: For a random model, the baseline ROC AUC is always 0.5. This reflects the fact that a random model produces a diagonal ROC curve (y = x), where the TPR equals the FPR across all thresholds, indicating no discriminatory power. Baseline PR AUC: For a random model, the baseline PR AUC equals the fraction of positive examples in the dataset. It is a good exercise to think about why the baseline PR AUC equals the fraction of positive examples in the dataset. So, for example, a model with ROC AUC = 90% and PR AUC = 70% can still be excellent for imbalanced datasets; the lower PR AUC just reflects the challenge of predicting the minority class accurately.

4

Building products using Machine Learning and data is still difficult. The tooling infrastructure is still very immature and the non-standard combination of data and software creates unforeseen challenges for engineering teams. What can be done ? - Hold your data scientists accountable using Science. - At a minimum, any AI/ML project should include an Exploratory Data Analysis, whose results directly support the design choices for feature engineering and model selection. - Data scientists should be encouraged to think outside-of-the box of ML, which is a very small box - Data scientists should be trained to use eXplainable AI methods to provide context about the algorithm's performance beyond the traditional performance metrics like accuracy, FPR, or FNR. - Data scientists should be held at similar standards than other software engineering specialties, with code review, code documentation, and architectural designs. Unlike Machine Learning, Data Science is not an academic discipline, with its own set of algorithms and methods. Reproducibility is a keystone of the scientific method, because it reinforces the validity of the discovery. “Science doesn’t care what you believe”. If you can’t reproduce the same result given the same experimental setting, then is it even real? https://lnkd.in/deZ8XeF7 #datascience #machinelearning #ai #artificialinteligence

4

Ever wondered how random numbers work in NumPy and why setting a random seed is important? 🌟 Check out my latest blog to learn more about true vs. pseudo-randomness and the role of random seeds. #Randomness #PseudoRandomness #NumPy #PythonProgramming #Algorithms #Computing #Cryptography #ScientificComputing

21

6 Comments

A common Databricks cluster misunderstanding is people think Spot instances will get them huge cost savings instantly... the reality is a bit more complicated... When Spot nodes are revoked randomly in a Spark cluster, the impact on the job's runtime can vary wildly. We've seen runtimes jump up 2-3X, even with just 1 worker revoked. This can cause the total cost of a job to even surpass "on-demand" prices! What can you do about this? Unfortunately Spot is really a random chaos machine and nobody has control over it (not even Databricks). Hopefully on average your costs are lower, but a static on-demand and optimized cluster can get you much more stable and possibly cheaper jobs. Be kind to your clusters folks! Sync Computing #dataengineering #AI

34

4 Comments

This is a major development for the open-source AI ecosystem. Apple has gone fully open-source! They have just released DCLM, a 7B open-source LLM, including weights, training code, and dataset. They plan to open-source all their models for deployment across their billion devices. Apple's embrace of open-source marks a significant shift. Here’s a summary: → Apple has released and open-sourced their DCLM 7B model, outperforming Mistral/Qwen2/Gemma on various benchmarks. → It is primarily trained on English data with a 2048 context window, and licensed under the Apple Sample Code License. → The release includes an exceptional explanation of the data curation process, offering valuable insights into LLM training. → DCLM is trained on 2.5T tokens using the DataComp-LM (DCLM) framework, with datasets from Common Crawl. → DataComp for Language Models (DCLM) is a testbed for controlled dataset experiments aimed at improving language models. → The DCLM Framework is designed for future experiments with data curation strategies like deduplication, filtering, and data mixing. Why is this significant? Apple's AI strategy now includes open-sourcing all their models. This shift, alongside their recent on-device AI announcements at WWDC, indicates a move toward smaller models and open AI. The future of AI is looking increasingly open. Resources: Model: https://lnkd.in/grgNegcN Repository: https://lnkd.in/gzcTsxqY Dataset: https://lnkd.in/gqY_U2x5

46

Let's keep the Snowflake vs Databricks beside for a moment. Please tell me more in the comments about your experiences using Dask and Spark. #dataengineering #bigdata #distributedcomputing

1

Sharing my intro Bayesian AB Testing readings and I welcome additional suggestions: Stucchio (2015) Bayesian A/B Testing at VWO https://lnkd.in/ghqekTZZ * The paper that started it all (for me). Stucchio offers Python code and a good explanation of stopping Bayesian AB Tests based on Expected Loss (AKA Risk). Paper is vague about how to arrive at an Expected Loss Threshold by saying let teams decide how much risk they can take ... something PMs and MMs freeze on in my experience. We developed an internal algorithm to compute an optimal risk threshold based on simulations after test owners supply values for max test sample, post test sample, and prior expectations for treatment and control variants. Harrell (2024) Efficacy design for experiments https://lnkd.in/gkkg-ccb * Not the easiest read for a newcomer, but the best case I’ve seen for adopting Bayesian operating characteristics if your primary goal is the correctness of your test decisions (which is different than minimizing incorrect decisions). Lakens (2017) Improving your Statistical Inferences https://lnkd.in/gcStguXy * First three modules are very engaging lectures on foundations of frequentist, likelihood and Bayesian statistics. Otsuka (2022) Thinking about Statistics https://a.co/d/0hp0NRLy * Wonderfully clear and engaging explanations of differences between frequentist, likelihood and Bayesian statistics. Bedtime reading. Berman (2020) Test & Roll: An experimental design to maximize profit from A/B Tests https://lnkd.in/gEn8CahV * Great framing of the difference between asking, how many samples do I need vs what's the best decision rule for the samples that I have. #bayes #abtesting

73

4 Comments

🎥 Just tested out Snowflake's new Cortex AI Analyst and pitted it against my own LangGraph GPT-4o agent—interesting results! 👀 Three key areas where I think Cortex falls short: 1️⃣ Stateless design – No memory capabilities, you can't ask follow on questions or refine your prompt. 2️⃣ Lack of independent reasoning – It’s more reactive than proactive. 3️⃣ Metadata stuffing – Handling large data warehouses with static semantic stuffing could lead to scaling and maintenance issues. 💡 But I was impressed by Cortex's ability to transform vague, ambiguous questions into something concrete, tailored to the data. My take: Cortex could thrive in a framework like LangGraph. It would solve the memory issue, allow more flexible metadata handling, and function better as a query-checking or ambiguity-resolution node, potentially in parallel with GPT. I imagine Cortex is fine-tuned for Snowflake, which means it might not play as well outside of it—but if your data lives in Snowflake, integrating Cortex within LangGraph could be a powerful combination! What are your thoughts? #AI #Cortex #Snowflake #LangGraph #OpenAI #SQL

13

3 Comments

PS. One of the best questions I heard today was around interrogating the weights/memory of your fine-tuned LLM. Does explainability matter? I honestly can't decide: Yes: You want to understand your model mechanics so that you can explain and predict it's behavior, especially on out-of-sample data No: This technology is so far beyond the NLP models we grew up with that explainability is no longer relevant

8

2 Comments

Ola Krutrim , India’s first AI unicorn, recently launched its LLM, Krutrim-7B-chat. This new model is trained in 10 Indian languages and is available on Databricks Marketplace. It can generate text in Hindi, English, Tamil, Telugu, Marathi, Malayalam, Gujarati, Bengali, and Kannada. This is the first of several India-specific models trained by Ola Krutrim. It has been trained on a massive dataset of Indic languages— “Krutrim delivers text infused with Indian sensibilities and cultural awareness.” Check out the model here. The announcement came to light at the Data Intelligence Day in Mumbai on May 10, 2024, by Gautam Bhargava, VP & Head of AI Engineering at Krutrim. “We’re excited to be at the forefront of building foundational LLMs trained on Indian languages to better serve our customers in India. We have been working very closely with the Databricks team to pre-train and finetune our foundational LLM,” said Bhargava. He added, “We are confident that Krutrim, our base model, will be able to provide high-quality responses in Indian languages and dialects.” According to Bhargava, Krutrim was trained using Databricks Mosaic AI Training’s GPU orchestration and system optimisations, enabling state-of-the-art generative AI solutions for customers globally. The model is now available on the Databricks Marketplace.

49

1 Comment

📊 Optimizing RAG Applications with TruLens: A Quick Guide Recently explored TruLens for evaluating and fine-tuning Large Language Models (LLMs) in Retrieval Augmented Generation (RAG) applications. Here's what I learned: Key Evaluation Metrics: ✅ Groundedness: Ensures responses are factually supported by context ✅ Answer Relevance: Measures if responses directly address queries ✅ Context Relevance: Evaluates quality of retrieved information The Process: Set up automated evaluation pipelines using TruLens Track performance across different configurations Use metrics to optimize: Chunking strategies Retrieval parameters Context window sizes Embedding models Benefits: Data-driven optimization Automated quality assessment Systematic performance tracking Clear metrics for improvement Result: More accurate, relevant, and reliable RAG applications with measurable improvements in response quality. #AI #MachineLearning #LLM #RAG #DataScience #TruLens https://lnkd.in/gHQnjREi

3

🚀 Exciting News in AI! 🌐 Snowflake AI Research has recently unveiled Arctic, an innovative open-source large language model (LLM) designed for enterprise applications. This breakthrough model boasts a staggering 480 billion parameters and utilizes a Dense-MoE Hybrid transformer architecture to efficiently manage tasks like SQL generation, coding, and instruction following. What sets Arctic apart is its cost-effectiveness and accessibility. Developed to operate under lower computing budgets, it allows businesses to adopt cutting-edge AI without the hefty price tag typically associated. The model is hosted on platforms such as Hugging Face, AWS, and NVIDIA AI Catalog, making it easily accessible and ready for enterprise integration. Snowflake Arctic is not only a testament to technological advancement but also a beacon for open-source collaboration. It is licensed under Apache 2.0, promoting transparency and community engagement. This strategic move is poised to spur further innovation and extend the model's capabilities through community-driven development. As businesses continue to seek competitive advantages, Arctic offers a scalable and adaptable solution that caters to diverse enterprise needs. Its release marks a significant leap forward in making advanced AI more feasible and impactful for businesses of all sizes. Stay tuned as we continue to follow the developments of this exciting project! #AI #MachineLearning #OpenSource #SnowflakeArctic #TechnologyTrends #BusinessInnovation https://lnkd.in/dZ3NkJgP

15

Going from PoC to Production for LLMs? 🤔 In the age of GPTs, chat interfaces are ubiquitous. But what if we add a voice/conversational layer on top? Imagine a C-suite member checking sales on the go or a new employee understanding leave policies through a natural voice interface. Intriguing, right? This got me thinking: how do we move from a PoC running on a local machine to a full-fledged production application? Here's what I found out: 1️⃣ Data Management and Knowledge Distillation: Handle and version your knowledge base (PDFs, Images, Excel, ER Diagrams). Keep data up-to-date and manage new information efficiently. How do you ensure information in the data is not outdated? What’s the best way to encode and chunk this data? The questions arise! 2️⃣ Access control and Citations: How do you ensure people cannot access data they are not supposed to have access to? How do you retrieve specific documents to ensure better answers (LLM models work the best when provided with specific information) and lower bills (Each LLM service charges by the number of tokens)? How do you remove any Personally Identifiable Information (PII)? What kind of caching strategies to use? (Pareto principle, 80% of user queries would be for 20% of the questions that get repeated) 3️⃣ Security and Reproducibility: Protect against bad actors and prompt injection. Create secure networks. How do you safeguard your application? It’s a growing field with evolving challenges. 4️⃣ Heart of the Operations (Sort of, I still think your knowledge base is the heart): Now that you have designed and taken care of the above aspects, there is still more. What size of the model to use? What environment to host it in? how to scale and replicate? (If you have a fine-tuned model). 5️⃣ Generating responses: How do you ensure the results generated are accurate and correct? While it is an active area of research, the only effective solution at this point is humans in the loop. Productionizing LLMs is complex, with more questions than answers. I’d love to hear your experiences and insights. What challenges have you faced, and what solutions have you found? If you can think of important things that I may have missed out, feel free to comment below! Ps. I will be writing and building more on this idea in the coming days, stay tuned for more :) #LLM #AI #MachineLearning #DataScience #VoiceInterface

12

1 Comment

LLMs are obliterating the Old Ways of how data scientists work. And no one's talking about it. It's NOT just code copilots. It's: - more flexible OCR - faster named entity recognition - fewer training samples to do classification - faster data labeling (with human-in-the-loop) - and even faster baselines for regression (Regression? Yes, regression! Check out "From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Example" on arvix for direct predictions on synthetic data. And "Boosting Tabular Data Predictions with Large Language Models" on Medium for an embeddings-as-features approach on a Kaggle problem's test set). LLMs are doubling data scientists' productivity by reducing time-to-value across a range of problems. With this comes new challenges: - PII leakage - prediction robustness - inference costs and speed - data leakage (e.g., from public datasets) There's an emerging playbook to handle these while delivering with speed: The Switcheroo. 1. start with LLMs on a "classic ML" problem to prove value fast without much/any feature engineering (avoid models hosted on the public cloud) 2. if predictions are performant and robust enough on an internal holdout set, deploy 3. as that generates data, train a smaller model to do the same thing 4. replace the LLM to cut ongoing costs (People have been doing this with vendors since the dawn of time. Now, we're treat the LLM as a pricey vendor that lets us get up-to-speed faster.) --- It's time to move beyond the copilot paradigm. And deliver on the promise of the sexiest job of the 21st century. It is the #datascience leader's job to think through the second-order implications of new technology developments. Your business leaders want these implications with respect to: - the Strategy (why) - the Products (what), and - the core Operating Model (how) But it's equally important to think through how it changes YOUR operating model. Meaning, how your team works, and how they can work better. The world is changing, and you must change with it. This stuff's exciting and is one of the reasons I chose to lead teams rather than staying an IC (without losing my technical skills). It is a hard transition. Soon, I'll be publishing my perspective on The Multidimensional Leader. It's a framework I wish I had 5 years ago when I started leading teams. I use it daily to balance the competing priorities of my leadership role. You'll want to do the same. Follow me so you don't miss out. Good luck, #thinksenior PS - a repost is appreciated if you think someone else will find this useful!

43

4 Comments