Houdong Hu

Hallucination is Inevitable in LLMs An important new paper out that shows that even with perfect data and an idealized architecture “LLMs will always hallucinate”. I suspect that this will over time be linked to Gödel’s Incompleteness Theorem [1] and Turing’s Halting Problem [2]. Every technology has trade-offs. We love LLMs but they need to be used in the right way. [1] Good pop science depiction in Oppenheimer (Gödel was Einstein’s walking companion) or also see Rebecca Goldstein’s book Incompleteness [2] Turing is depicted in The Imitation Game and also the progenitor of the famed Turing Test.

36

Good detailed overview of Retrieval-Augmented Generation (RAG), one of leading techniques to improve accuracy of LLMs. When developing a production-grade LLM-based system, it's essential to understand the advantages and limitations of different techniques (RAG, prompt engineering, fine-tuning etc), across training and serving/inference phases, to improve LLMs for specific applications (see my previous post https://lnkd.in/giuJdJJt) A balanced approach to LLM applications that takes into account accuracy, latency, and cost is crucial for successful productization at scale. #artificialintelligence #largelanguagemodels #GenerativeAI #airevolution

10

Learn how embedding models are built, trained, and used in semantic search systems in this new short course, Embedding Models: From Architecture to Implementation, created with Vectara and taught by Ofer Mendelevitch. Many LLM apps use a single embedding model for both questions and answers. This can lead to issues such as retrieving results that are similar to the question itself rather than relevant answers. With a dual encoder architecture, you can use separate embedding models for questions and answers to better match questions with appropriate answers. In this course, you will use, build, and train a dual encoder model. You’ll also learn: - What are word embeddings, and how they are used - The evolution of embeddings to BERT, where embeddings take into account each word's surrounding context - How a contrastive loss is used to train a dual encoder model with one encoder trained to embed questions and the other responses. - How to analyze a dual encoder’s effect on search relevance and compare it to a retrieval process with a single encoder. Please sign up here! https://lnkd.in/gvYghsxV

3,365

83 Comments

Spatial Reasoning did not ‘emerge’ spontaneously in Large Language Models (LLMs) the way so many reasoning capabilities did. Humans have specialized, highly capable spatial reasoning capabilities that LLMs have not replicated. But every subsequent release of the major models — GPT, Claude, Gemini- promises better multimedia support, and all will accept and try to use uploaded graphics along with texts. #llms #chatgpt4 #gemini #llmops #aiml #ml https://lnkd.in/dFSWKqKs

2

Awesome to see the Meta #Llama 3.2 models on #Azure #AI Model Catalog through managed compute inferencing #MaaS and soon on serverless API deployments while MACC Eligible providing customers with Enhanced Security and Compliance, Content Safety Integration, Simplified Assessment of LLM flows, Client integration, and Simplified Deployment and Inference. https://lnkd.in/g4Zpb3Dm

9

1 Comment

Can LLM preference tuning degrade Multi-Agent system performance? In a recent tweet, Andrej Karpathy questioned why all LLMs sound similar, prompting Sebastian Rascha to speculate about the uniformity of alignment datasets. Aligning user preferences is a last-mile step in training large language models. Regardless of the pre-training datasets used, could the preference datasets used during alignment compromise the effectiveness of LLMs? This issue could adversely affect multi-agent systems incorporating multiple LLMs in their design.

7

1 Comment

🔍 Key Papers to Demystify How GPT-o1 Works 1) Training Language Models to Self-Correct via Reinforcement Learning: https://lnkd.in/gwgdE5fb 2) STaR: Bootstrapping Reasoning With Reasoning: https://lnkd.in/gRdir7PP 3) Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking: https://lnkd.in/g47fqB7p 4) V-STaR: Training Verifiers for Self-Taught Reasoners https://lnkd.in/gGRRyBrF 5) Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters: https://lnkd.in/g9atmpvC 6) Chain of Thought Empowers Transformers to Solve Inherently Serial Problems: https://lnkd.in/gCyMncsf 7) Large Language Monkeys: Scaling Inference Compute with Repeated Sampling https://lnkd.in/ghy2r5nH 8) An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models: https://lnkd.in/gRHU2xyU 💡 Stay Tuned! I will continually update this post with more insights and a special focus on the "how" aspects of these papers.

18

2 Comments

LLMs and Optimization: An Unbeatable Combo!

19

The game of large language models (LLMs) got a major plot twist! Meta's unveiling of Llama, a family of AI models pushing the boundaries of parameter size and processing power, is a game-changer. Why it matters? Llama boasts a staggering 405 billion parameter model, dwarfing previous heavyweights like Google's LaMDA and OpenAI's Jurassic-1 Jumbo. This sheer size unlocks unprecedented capabilities in: -> From scientific papers to philosophical treatises, Llama can digest vast amounts of data, leading to deeper comprehension. -> Imagine writing styles that perfectly mimic different authors or create compelling narratives on the fly. -> From machine translation to chatbot interactions, Llama holds the potential to transform how humans and machines communicate. -> Llama isn't just about brawn. Meta stresses its commitment to open-source access and responsible AI development. This fosters collaboration and helps mitigate potential biases often associated with large datasets. Meta's ambitious project is a harbinger of the future, where AI can not only process information but truly understand it. Imagine personalized education tailored to individual learning styles, or AI assistants that anticipate your needs before you even ask. https://lnkd.in/dYn_cTjB #MetaLlama #ai #LLMs #futureoftechnology #naturallanguageprocessing #opensource

6

Ensemble of smaller LLMs (PoLL) is less biased, faster, 7x cheaper when evaluating LLMs performance. Works great on QA & Arena-hard evals arxiv.org/abs/2404.18796

9

Learn how to build an autonomous research assistant using LangGraph with a team of specialized AI agents #llms #chatgpt4 #gemini #aiml #llmops https://lnkd.in/d6TUbZNj

2

Learn how to get up and running with Claude 3.5, the newest model from Anthropic on #VertexAI with Weights & Biases Weave! This tutorial is designed to help you get started with the Claude 3.5 Sonnet model on Google Cloud’s Vertex AI using Python. #aiml #generativeai #googlecloudpartners

8

🚀 Does Size Always Matter? 🧐 When it comes to Large Language Models (LLMs), the answer isn’t always straightforward. While larger models often top leaderboards with impressive average benchmark scores, real-world applications tell a different story. A few considerations: • 📊 Leaderboards favor larger models, but these models might not always be the best for specific tasks. • 🎯 For business applications, we often use LLMs for single tasks or a narrow subset of tasks, which align with specific benchmarks, not the whole portfolio. • 💡 A great example is Bielik 2 - a Polish LLM with just 11B parameters, performing well in SpeakLeash.org Arena, where it ranks between LLaMA 3.1 405B and 70B (this ranking reflects how well the responses of different models meet the subjective expectations of Polish users). • 💸 This becomes especially important when the models under consideration differ significantly in terms of cost. Key Takeaway: When choosing an LLM, prioritize benchmarks or rankings that closely match your use case over general leaderboards, or even create your own benchmarks closely aligned with your needs. #AI #MachineLearning #LLM #BusinessInsights #TechTrends #AIModels

52

5 Comments

Organizations have insufficient technical capabilities for the speed and scale of their ML model implementations. My colleague Sumit Agarwal just published a very helpful research with a capability view and sample vendors for defining a modular, flexible and scalable MLOps solution. https://lnkd.in/gnB_T9bq #MLOps #LLMOps #AI

149

17 Comments

Open-source AI for evaluating language models Researchers from KAIST AI, LG AI Research, CMU, MIT, Allen Institute for AI, and UIC have introduced Prometheus 2, a groundbreaking open-source evaluator model that assesses language model outputs with accuracy rivaling expensive proprietary solutions like GPT-4. A few key highlights: Prometheus 2 combines models trained on direct assessment and pairwise ranking, enabling it to excel at both common evaluation formats. It leverages the new Preference Collection dataset with 1,000 evaluation criteria to boost performance. Prometheus 2 achieved the highest correlation with human judgments and proprietary models across 8 benchmarks, in some cases reaching 0.90 Pearson correlation and 85%+ accuracy. This helps narrow the gap between open-source and closed-source AI evaluation without sacrificing transparency and control As an AI and Data Strategist, I like that this will make high-quality AI evaluation more accessible and scalable for researchers and practitioners. Some potential strategic impacts: Accelerating AI/ML system development by enabling rapid, reliable evaluation of generated language outputs at lower cost Establishing clearer standards and benchmarks for assessing language model performance as open alternatives mature #ai #machinelearning #nlp #opensource #research #aistrategy Paper - https://lnkd.in/gsnGDwJd

1

Interesting look at evaluating AI applications quality for production deployments. Luís Roque unpacks Relative Answer Quality (RAQ) a technique looking at question answers using an independent LLM. Once evaluated, he shows how NIM can help deploy these models at scale. https://lnkd.in/dBcMsk6U #generativeAI #NVIDIANIM #AIinference

34

1 Comment

These findings motivate our design of a novel framework to investigate hidden representations in LLMs with LLMs, which we call Patchscopes. The key idea behind this framework is to use LLMs to provide natural language explanations of their own internal hidden representations. Patchscopes unifies and extends a broad range of existing interpretability techniques, and it enables answering questions that were difficult or impossible before. For example, it offers insights into how an LLM's hidden representations capture nuances of meaning in the model's input, making it easier to fix certain types of reasoning errors. While we initially focus the application of Patchscopes to the natural language domain and the autoregressive Transformer model family, its potential applications are broader. For example, we are excited about its applications to detection and correction of model hallucinations, the exploration of multimodal (image and text) representations, and the investigation of how models build their predictions in more complex scenarios. Google DeepMind

9

AtScale Open-Sourced Semantic Modeling Language (SML): Transforming Analytics with Industry-Standard Framework for Interoperability, Reusability, and Multidimensional Data Modeling Across Platforms AtScale has made a significant move by announcing the open-source release of its Semantic Modeling Language (SML). This initiative aims to provide an industry-standard semantic modeling language that can be adopted across various platforms, fostering greater collaboration and interoperability in the analytics community. The introduction of SML marks a major step in the company’s decade-long journey of democratizing data analytics and advancing semantic layer technology. SML offers several benefits to developers and organizations: 🔰 Object-Oriented Structure: SML is designed to be object-oriented, so its semantic objects can be reused across different models, promoting consistency and efficiency in model building. 🔰 Comprehensive Scope: It is a superset of existing semantic modeling languages, incorporating more than a decade’s experience and use cases across different verticals. This makes SML versatile enough to cater to a wide range of applications. 🔰 Familiar Syntax: SML is built on YAML, a widely adopted, human-readable syntax, making it easier for developers to adopt the language without steep learning curves. 🔰 CI/CD Friendly: Being code-based, SML integrates well with modern software development practices, including Git for version control, and supports continuous integration and continuous deployment (CI/CD) workflows. 🔰 Extensibility and Open Access: SML is Apache open-sourced, which means it is free to use and can be extended by the community. This open nature allows for innovation and collaboration, ensuring the language evolves to meet new demands. Read our full take on this: https://lnkd.in/geYKRkcA GitHub: https://lnkd.in/gabaJSDx Details: https://lnkd.in/gRNeAVic AtScale David P. Mariani Bri Pedersen Rouzbeh Safaie Jeffrey Curran #opensource #ai

44

1 Comment