Haifeng Zhao’s Post

Whether you're selecting models, fine-tuning, or moving to production, having a robust evaluation strategy is important for LLM applications. In this new article we discuss various strategies for figuring out if an LLM application is working correctly (or not), covering: * Scaling evaluation from manual to automated testing * Data sourcing strategies including 'Correct by Construction' * Key metrics from traditional NLP to LLM-as-Judge approaches * Practical implementation considerations https://lnkd.in/gsSEqdZQ Thanks to my co-authors Angeline Yasodhara and Rodrigo Ceballos Lentini #LLM #AI #MLOps #SoftwareEngineering #GenerativeAI

Level up your LLM game: Choosing the right metrics for success Building effective Large Language Models (LLMs) requires a data-driven approach. But choosing the right metrics to evaluate success can be tricky! This post breaks it down for you: Why it matters: Measuring LLM performance is crucial, whether you're in research or production. It helps you: * Deliver the user experience you envisioned (production) * Validate your LLM's capabilities (research) Picking your champions: The best metrics depend on your LLM's purpose: * Classical tasks (classification): Look to libraries like torchmetrics and sklearn.metrics. * Generative tasks: * RAG: RAGAS library is your friend! * Code generation: Evaluate both execution accuracy and efficiency. * Limited ground truth? Build an LLM evaluator on a smaller dataset for broader assessment. Want to dive deeper? Stay tuned for a comprehensive metrics list by task category! #LLMs #MachineLearning #AI #Metrics P.S. Feel free to share your favorite LLM evaluation tricks in the comments! https://lnkd.in/gMagVxna

Into the world of Large Language Models (LLMs) it's fascinating to see how much goes into optimizing them for performance and scalability. Techniques like prompt engineering, model pruning, and even load balancing are essential to make sure these models run efficiently in real-world applications. It’s incredible how much power and potential these models have, but it also shows me just how important it is to master these optimization strategies. Learning this stuff isn’t just about building something cool—it’s about making sure it works at scale. #AI #MachineLearning #LLMs #DeveloperLife https://lnkd.in/epHddQZg

Unleash the Power of LLMs: Tailoring General Models for Specialized Tasks Fine-tuning Large Language Models is the key to optimizing AI performance for domain-specific tasks, from legal contracts to sentiment analysis. #AI #LLM #Finetuning https://lnkd.in/dsd-H5T8

"Generative AI and large language models (LLMs) like GPT-4, Llama, and Claude have pathed a new era of AI-driven applications and use cases. However, evaluating LLMs can often feel daunting or confusing with many complex libraries and methodologies, It can easily get overwhelming. LLM Evaluation doesn't need to be complicated. You don't need complex pipelines, databases or infrastructure components to get started building an effective evaluation pipeline." https://lnkd.in/dk9FiAVH.

The supply of quality, real-world data used to train generative A.I. models appears to be dwindling as digital publishers increasingly restrict their access to their public data. That means the advancement of large language models like OpenAI’s GPT-4 and Google’s Gemini could hit a wall once the A.I.s scrape all the remaining data on the internet. To address the growing A.I. training data crisis, some experts are considering synthetic data as a potential alternative. Read more: https://lnkd.in/exifvztU By Aaron Mok

Selective State Spaces (SSMs) offer a promising alternative to large language models like GPT, addressing key challenges such as computational inefficiency with long sequences and high energy use. SSMs efficiently process long data sequences and focus on the most relevant information, making them faster and more resource-efficient. This adaptability makes SSMs ideal for tasks like legal document review, e-discovery, and legal research. Models like Mamba, which leverage SSMs, are looking like they outperform traditional models in handling extensive data, offering significant improvements in performance and practicality for legal tech applications. Have a deeper read about SSMs in my recent blog post: #ssm #llms #legaltech #artificialintelligence

https://lnkd.in/dP9YXYp7 This article explains how to use an LLM (Large Language Model) to perform the chunking of a document based on concept of “idea”. I use OpenAI’s gpt-4o model for this example, but the same approach can be applied with any other LLM, such as those from Hugging Face, Mistral, and others. Everyone can access this article for free.....

🔆 Exciting approach to evaluating LLMs on factuality DeepMind's Search-Augmented Factuality Evaluator (SAFE): 🖌 Automated evaluation: Employs an LLM to assess the factuality of long-form text generated by LLMs. 🖌 Fact verification: Breaks down text into individual claims and uses Google Search to verify their accuracy. 🖌 Independent operation: Reduces the reliance on human annotators for evaluating LLM outputs. 🖌 Higher agreement rate: Demonstrates superior agreement with human judgments compared to individual human annotators. 🖌 F1@K metric: Extends the traditional F1 score to measure the overall factuality of long-form responses, balancing precision and recall based on desired length. 🖌 Open access: LongFact dataset and SAFE code available on GitHub for further research and development. 🖌 Potential for LLMs: Showcases the ability of LLMs to not only generate content but also evaluate and improve the quality of their own outputs. You can read more about it here - https://lnkd.in/gcXrzzJ6 #LLM #GenAI #AI #ML #Learning #Datascience

How to Interpret GPT2-Small - Mechanistic Interpretability on prediction of repeated tokens by Shuyang Xiang