Taranveer Singh

Most (maybe all?) LLM performance benchmarks like Artificial Analysis go in depth on *online* inference. *Batch* inference seems simpler since almost all companies run some form of embarrassingly parallel workloads. But batch inference is different from other map-reduce style workloads. The biggest differences are the use of GPUs and often mixed CPU + GPU compute. Today, tons of energy is going into improving systems for batch inference, and batch inference is just as important as online inference (companies have accumulated tons of data over the years, and if you want to use AI to extract insights or make decisions based on that data, you will need to process that data in a batch fashion). A bunch of companies are speaking at #RaySummit on their best practices for batch inference: Apple: "petabyte scale embedding generation" Roblox: "efficient batch inference at Roblox" Uber: "large scale generative AI batch prediction with Ray and vLLM" eBay: "pioneering next-gen AI platform" Handshake: "job description parsing with batch LLM inference" https://lnkd.in/deWkzHEC

260

4 Comments

The AI Model Race: A Prank Gone Out of Control The naming of LLMs these days feels like that infamous file name: Report_2024_latest_v5_final_fixed.docx. And now there’s a trend of releasing models with date-stamped versions. Essentially, it’s the same model, just slightly better. For example: on November 14, Google released Gemini-Exp-1114, which immediately took first place on Chatbot Arena. Then, on November 20, OpenAI dropped ChatGPT-4o-latest (2024-11-20), reclaiming the top spot. But it seems Google was just waiting for this. The very next day (November 21), they released Gemini-Exp-1121, taking first place again. Clearly, they didn’t train it overnight. As a result, the leaderboard is now crowded with models that have intimidating names and nearly identical scores. … An OpenAI researcher walks into a café and looks at the menu. “Avocado Toast,” “Avocado Toast 2.0,” “Mini Avocado Toast,” “Gourmet Super-Toast,” “Experimental Charcoal-Grilled Avocado Toast,” and finally, “Truffle Avocado Toast for Premium Subscribers.” - So where’s the leaderboard? - asks the researcher.

14

1 Comment

I can sense the LLM-hype cycle crashing soon. Don’t forget: 1. End-user value matters. Just adding LLM doesn’t make your product better 2. LLMs can’t plan/reason; no amount of monkey patching will overcome this limitation 3. RAG isn’t always better than vanilla BM25 search processed 200+ applications for the AI hackhouse. Many seem to not understand the basic limitations of LLMs + everyone seems to be building similar wrappers on openAI API. I think it’s a function of perverse incentives: it is profitable to hype up your product’s capabilities Ultimately, users don’t care about AI or LLMs. What matters is if you’re solving their problems in a significantly better way v/s all other alternatives. (And remember now all other alternatives are also LLM-powered).

176

15 Comments

🌟 Day 4 of My 25-Day Generative AI and LLM Challenge: How Large Language Models (LLMs) Works 🌟 with my friends Maanan Jagani and HARSH JAVIA! Large Language Models (LLMs) like GPT-3 and BERT have revolutionized the field of Generative AI. Today, we'll dive into the architecture and training process that make these models so powerful. 🧠 Architecture of LLMs 1. Transformer Architecture: LLMs are built on the transformer architecture, introduced in the paper "Attention is All You Need" by Vaswani et al. (2017). "Attention is All You Need" : https://lnkd.in/d3CV4xvz Key Components: Self-Attention Mechanism: Allows the model to weigh the importance of different words in a sentence when encoding a particular word. This helps in capturing long-range dependencies in the text. Feed-Forward Neural Networks: After the self-attention layer, the data is passed through a feed-forward network for further processing. Positional Encoding: Since transformers do not inherently understand the order of words, positional encodings are added to provide this information. 2. Layers and Parameters: LLMs consist of multiple layers of transformers. Each layer contains self-attention and feed-forward sub-layers. Parameters: LLMs like GPT-3 can have hundreds of billions of parameters, making them capable of understanding and generating complex language patterns. 🧠 How Do LLMs Work? 1. Data Collection: LLMs are trained on massive datasets, including books, articles, websites, and more. This diverse data helps them grasp the nuances of human language. 2. Neural Network Architecture: The core of LLMs is the Transformer architecture, which uses self-attention mechanisms to process input data. This allows the model to understand relationships between words in a sentence, regardless of their position. 3. Tokenization: Text is broken down into smaller units called tokens. Tokens can be as short as a single character or as long as a word or phrase. This breakdown helps the model manage and analyze the text more effectively. 4. Training Process: During training, the model learns to predict the next word in a sentence by analyzing the context provided by the previous words. This process involves adjusting millions or even billions of parameters to minimize prediction errors. 5. Fine-Tuning: After the initial training, LLMs can be fine-tuned on specific tasks or domains, such as medical text or legal documents, enhancing their performance in these areas. More on Fine-Tuning: https://lnkd.in/dpjS5Jhw 6. Inference: Once trained, LLMs can generate text by taking an initial prompt and predicting subsequent words, creating human-like responses or content. You can go and check out the video on how LLMs works by IBM Technology: https://lnkd.in/dWHz6G3n Curious to see LLMs in action or learn more? Drop a comment or send a message! #AI #LLMs

12

📢 Exciting Research Alert! 🚀 📚 Just came across a groundbreaking paper titled "Linearizing Large Language Models" by Jean Mercat, Kushal Arora, Achal Dave, Igor Vasiljevic, Adrien Gaidon, Sedrick Keh, and Thomas Kollar from Toyota Research Institute. This paper introduces a novel approach called Scalable UPtraining for Recurrent Attention (SUPRA) to convert pre-trained language models into efficient Recurrent Neural Networks (RNNs). Let's dive into the key research points! 🔬 🔹 The paper addresses the challenge of inference cost in Transformers, which scales linearly with the number of tokens. To overcome this, the authors propose a linearization technique that converts pre-trained Transformers into RNNs, reducing the cost of inference. This technique leverages the strong pre-training data and performance of existing Transformer models while requiring only 5% of the training cost. 🔹 The authors introduce the concept of linear attention, which replaces the softmax dot-product attention with a more generic similarity function. They explore different similarity functions, including a linear kernel, and demonstrate the computational benefits of linear attention. This allows the models to be trained with sequence parallelism (as Transformers) but operate as RNNs at inference time. 🔹 The paper discusses the limitations of recurrent LLMs (Linear Language Models) compared to Transformers, particularly in in-context learning and long-context tasks. The authors compare pre-trained and uptrained RNNs to Transformers, revealing a persistent gap in performance for these tasks. 🔹 The methodology section provides a detailed explanation of linear attention, recurrent inference, and the process of finetuning a Transformer into an RNN. The authors build upon previous work and propose modifications to address stability issues and improve performance at larger scales. 🔹 The paper also highlights the importance of using modern relative positional encoding schemes, such as RoPE, for competitive performance in linear attention models. 🔗 For those interested in exploring the code and models, the authors have provided a GitHub repository: [GitHub Link](https://lnkd.in/eAivZekb). This repository contains the code and models related to the SUPRA approach, allowing researchers to replicate and build upon the findings of the paper. 📝 In summary, "Linearizing Large Language Models" presents a cost-effective alternative to pre-training linear Transformers by uptraining existing models into efficient RNNs. The paper explores the mathematical aspects of linear attention, recurrent inference, and the SUPRA methodology. This research opens up new possibilities for reducing inference cost and improving performance in language and multimodal models. Don't miss out on this groundbreaking work! 🌟 #ResearchPaper #LanguageModels #Transformers #RNNs #SUPRA #MachineLearning #NLP #ArtificialIntelligence #GitHub

2

What happened to OpenAI? Back in 2015-2018, there was a quirky group of non-profit startups in SF all trying to do things differently with tech: Code for America, DataKind, Bayes Impact (which I led for a time). And, of course, OpenAI. There was so much hope and energy around using tech for good.  Over time, OpenAI evolved into a for-profit company, but its core DNA seemed intact. I saw them as a developer tool and infrastructure company, filled with amazing talents like Karpathy and Ilya, aimed at empowering developers to build remarkable products. They seemed focused on the next frontier of AI, trusting the broader developer community to connect with end users. But I wonder if the launch, and unbelievable success, of ChatGPT completely derailed their mission.  I miss the nerdy OpenAI—the one where a group of brilliant nerds built for a wider community of brilliant nerds. OpenAI doesn't need to mimic Scarlett Johansson's voice to be cool. Being the AI infrastructure of the world is incredibly cool! Please, stay cool. #AI #TechForGood #OpenAI https://lnkd.in/epgy35VW

19

💡 How to Build an Experimentation Machine and Where Most Go Wrong with Ramesh Johari (Stanford, Airbnb, Uber, Bumble, and more) 🎙️ In this episode of High Signal, I talk to Ramesh Johari—Stanford professor and advisor to platforms like Airbnb, Uber, and Bumble—about building systems for experimentation that drive innovation and avoiding the traps most organizations fall into. We explore practical takeaways like:  - ⚡ How to combat risk aversion by running faster, smaller tests that reduce stakes while increasing learning.  - 🔄 Why a successful experimentation strategy requires learning across tests instead of treating experiments as one-offs.  - 🌟 How to create a self-learning organization, where experimentation fuels continuous improvement and innovation. 🎬 Watch the clip where Ramesh shares how organizations can build a "learning flywheel" to stay adaptive in dynamic environments. Produced by Delphina with Duncan Gilchrist and Jeremy Hermann, High Signal brings you deep, actionable conversations to help advance your career in data science, AI, and machine learning. 💻 Check out the full episode via the link in the comments!

38

7 Comments

If only you could have fixed hallucinations of a pre-trained LLM through contextually relevant prompts. It would have been a solved problem by now! Contextual relevance (RAGs) improve domain specificity, not hallucination! Nice article. #llms #chatgpt4 #aiml #mlops #llmops #ml #rag https://lnkd.in/d44hdifr

11

SimpleQA: Measuring Hallucinations in LLMs Yesterday, OpenAI released a new benchmark for evaluating model hallucinations. (Surprisingly, it’s open-source from OpenAI!) Rather than dig into the benchmark specifics, I’ll focus on some interesting takeaways and observations. I liked that hallucinations are measured through the concept of model calibration. Calibration is the process of evaluating and adjusting a model so that its confidence in its answers aligns with actual accuracy. For instance, if a model claims 75% confidence in an answer, ideally, the accuracy of its responses should also be 75% in such cases. This lets us assess whether a model “knows what it knows” — essentially, how well it evaluates its own knowledge and confidence level. Upon analyzing calibration, it’s apparent that the more confident a model is in an answer, the more accurate the answer actually is. But here’s the bad news: models (so far, all of them) consistently overestimate their confidence. So, while confidence gives a rough estimate of accuracy, it isn’t entirely reliable. In short, use it carefully. Another finding: larger models hallucinate less. GPT-4o scores higher than GPT-4o-mini, just as o1-preview outperforms o1-mini. This was somewhat surprising to me. Takeaway: one way to reduce hallucinations is to make models even larger. One calibration measurement method is to ask the model the same question 100 times. From there, you can measure the frequency of a specific answer relative to its correctness (accuracy). It turns out that the more frequent an answer is, the more likely it is to be accurate. In other words, the most common answer is the most correct. Takeaway: we get a wide range of responses when a model’s accuracy for a given question is low. This aligns with what we might intuitively expect. Another interesting point is that the benchmark uses three types of evaluation: correct, incorrect, and not attempted. The first two are clear, but the last one refers to cases where the model responds with something like, “I don’t know the answer to this question.” Here’s the intriguing part: the o1 family uses this response type more often than the GPT-4o family. The authors speculate this is due to o1’s built-in reasoning process, though this remains just a hypothesis for now. https://lnkd.in/eFsz3-6P

13

2 Comments

Quentin H. and Yanwei (Wayne) Zhang on the Faire Data team recently did some great work fine-tuning Llama3 model to enhance search relevance. Here are some key insights from the work: • While prompt engineering is valuable, LLM fine-tuning is essential for a domain specific task like understanding semantic search relevance. • Fine-tuned open-source LLMs like Llama3 can deliver impressive performance, scalability, and cost-efficiency compared to using proprietary models. • Effective application of a fine-tuned LLM demands a well-defined problem, robust, high-quality labeled data, and ongoing model refinement. Ground-truth labeled data is crucial for maximizing model performance. • Fine-tuned Llama3-8b achieves performance on par with the larger Llama2–13b model. Read the blog post to dive deeper into these findings! https://lnkd.in/ecP_3E7S We are also hiring (https://lnkd.in/dHfRZcPi) #LLM #AI #Finetuning #LLAMA3 #SearchRelevance #Hiring

93

3 Comments

Prompt engineering is a rapidly-evolving topic in AI, but recent research can be grouped into four categories... (1) Reasoning: Simple prompting techniques are effective for many problems, but more sophisticated strategies are required to solve multi-step reasoning problems. - [1] uses zero-shot CoT prompting to automatically generate problem-solving rationales to use for standard CoT prompting. - [2] selects CoT exemplars based on their complexity (exemplars that have the maximum number of reasoning steps are selected first). - [3] improves CoT prompting by asking the LLM to progressively refine the generated rationale. - [4] decomposes complex tasks into several sub-tasks that can be solved via independent prompts and later aggregated into a final answer. (2) Tool Usage: LLMs are powerful, but they have notable limitations. We can solve many of these limitations by teaching the LLM how to leverage external, specialized tools. - [5, 6] finetune a language model to teach it how to leverage a fixed, simple set of text-based APIs when answering questions. - [7] uses a central LLM-based controller to generate a program—written in natural language—that composes several tools to solve a complex reasoning task. - [8] uses a retrieval-based finetuning technique to teach an LLM to adaptively make calls to APIs based on their documentation when solving a problem. - [9] uses an LLM as a central controller for leveraging a variety of tools in the form of deep learning model APIs. - [10, 11] integrates code-capable LLMs with a sandboxed Python environment to execute programs when solving problems. (3) Context Window: Given the emphasis of recent LLMs on long contexts for RAG / few-shot learning, the properties of context windows and in-context learning have been studied in depth. - [12] shows that including irrelevant context in the LLM’s prompt can drastically deteriorate performance. - [13] finds that LLMs pay the most attention to information at the beginning/end of the prompt, while information placed in the middle of a long context is forgotten. - [14] proposes a theoretically-grounded strategy for optimally selecting few-shot exemplars. (4) Better Writing: One of the most popular use-cases of LLMs is for improving human writing, and prompt engineering can be used to make more effective writing tools with LLMs. - [15] improves the writing abilities of an LLM by first generating an outline and then filling in each component of the outline one-by-one. - [16] uses a smaller LLM to generate a “directional stimulus” (i.e., a textual hint) that can be used as extra context to improve an LLM’s writing ability on a given task. - [17] improves the quality of LLM-generated summaries by iteratively prompting the LLM to increase the information density of the summary.

221

8 Comments

"Framing the AI" A recent paper from CMU by Behnam Mohmmadi explored cognitive bias in Large Language Models (LLMs). The paper is titled:- “Wait, It’s All Token Noise? Always Has Been: Interpreting LLM Behavior Using Shapley Value”. Understanding LLM’s cognitive behavioral tendencies is essential for marketers for two key reasons: 1️⃣ It helps harness the full potential of LLMs in production. For instance, an LLM sensitive to the "framing effect" may require careful prompting in recommendation systems. 2️⃣ LLMs with human-like behaviors serve as valuable test environments for consumer behavior experts, enabling preliminary studies before conducting main experiments. Consider the framing effect, introduced by Kahneman and Tversky in 1981. They demonstrated that individual choices are heavily influenced by how problems are framed. The paper tests this with LLM (lama-2-70B-Chat), using a prompt about flight choices. By positively framing one option (adding "only" before the price), we observe a significant shift in the LLM's preferences, mimicking human behavior. Here is the link to the paper: https://lnkd.in/gAd8dtcT This sensitivity suggests LLMs may reflect learned statistical associations rather than genuine cognitive processes. However, using Shapley values, we can identify key tokens influencing decisions, aiding in prompt optimization to mitigate apparent cognitive biases. The findings from the paper highlight the practical applications of Shapley value analysis in marketing, ensuring more reliable and consistent LLM performance in real-world scenarios. If you are interested in learning how to apply Shapley additive methods to explain the inner workings of ML models, please check out the free Explainable AI (XAI) course on Vizuara’s Youtube channel: https://lnkd.in/gNEx3ghr

38

1 Comment

Thanks Werner Bogula for the really insightful study. I’m only just getting into model comparison, for now using HuggingChat and some tightly defined use cases. The difference between them is notable, interesting to see, and makes me think the future is probably going to be based around multi-LLM workflows. #genai #multillm

1

Perhaps the most critical thing in a Retrieval Augmented Generation (RAG) system is the retrieval or searching strategy. Poor retrieval will lead to irrelevant documents going to the LLM leading to irrelevant generated answers or worse hallucinations. Is cosine similarity everything? Do check out below illustration which talks about four of the most common strategies for playing around with retrieval strategies in RAG systems: 1. Similarity-based retrieval: This is based on embedding similarity, you can play around with different metrics like cosine, maximal margin relevance etc. 2. Similarity with Threshold retrieval: Useful where we cap the similarity threshold and can help in removing irrelevant document but can drop recall 3. Multi-query retrieval: Make the LLM generate variations of your query and get documents based on these multi-versions of your initial query, often useful to get more recall or coverage of documents 4. Reranker retrieval: Use a model specialised in ranking documents based on search relevance Often you can combine some of these techniques and even use ensemble retrievers. Do explore them and go beyond just cosine similarity!

17

🔥 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈 𝐚𝐧𝐝 𝐋𝐋𝐌 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 - 𝐓𝐨𝐩 𝐪𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬 🔥 🤖 I've conducted a mock interview with my student, Bheesetti Anand, focused on Generative AI and LLM (Large Language Models) interview questions. This session is designed to benefit other students preparing for similar interviews, helping them secure their dream roles in the field of Generative AI. 📔 Here’s why watching the video can help you make a 𝐬𝐮𝐜𝐜𝐞𝐬𝐬𝐟𝐮𝐥 𝐜𝐚𝐫𝐞𝐞𝐫 𝐬𝐰𝐢𝐭𝐜𝐡 𝐭𝐨 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈: ✅ 𝐂𝐨𝐦𝐩𝐫𝐞𝐡𝐞𝐧𝐬𝐢𝐯𝐞 𝐂𝐨𝐯𝐞𝐫𝐚𝐠𝐞: Detailed exploration of top interview questions related to Generative AI and Large Language Models. ✅ 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐈𝐧𝐬𝐢𝐠𝐡𝐭𝐬: Real-world examples and practical applications to understand complex concepts better. ✅ 𝐄𝐱𝐩𝐞𝐫𝐭 𝐆𝐮𝐢𝐝𝐚𝐧𝐜𝐞: Tips and strategies to approach and answer interview questions effectively. ✅ 𝐂𝐚𝐫𝐞𝐞𝐫 𝐏𝐫𝐞𝐩𝐚𝐫𝐚𝐭𝐢𝐨𝐧: Essential knowledge and skills to prepare for interviews and stand out to potential employers. 📽 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 𝐥𝐢𝐧𝐤: https://lnkd.in/g_nkP7fs #generativeai #llm #interview #jobpreparation #mockinterview

14

2 Comments

"Schrödinger's Memory: Unveiling the Hidden Mechanics of AI Cognition" What if AI memory worked like Schrödinger's cat? A fascinating new research paper explores this intriguing concept, shedding light on the hidden mechanics of Large Language Models (LLMs) and their surprising similarities to human cognition. 👉 Schrödinger's Memory in LLMs The paper introduces the concept of "Schrödinger's Memory" in LLMs: - LLMs exhibit memory-like capabilities without traditional storage mechanisms - This memory only becomes observable when queried, much like Schrödinger's famous thought experiment - To understand this, imagine a box containing both a cat and a vial of poison. In quantum mechanics, the cat is simultaneously alive and dead until observed. Similarly, LLM memory exists in a superposition of states until queried This challenges our conventional understanding of AI memory systems and opens up new avenues for research. 👉 Universal Approximation Theorem (UAT) and LLMs The researchers apply the Universal Approximation Theorem to explain LLMs' memory capabilities: - UAT demonstrates how LLMs dynamically fit outputs to inputs - This theoretical framework provides insights into LLMs' ability to generate contextually relevant responses - Think of LLMs as master impersonators, able to mimic any function or behavior based on the input they receive 👉 LLMs vs. Human Memory: Unexpected Parallels One of the most intriguing aspects of this research is the comparison between LLM and human memory mechanisms: - Both LLMs and human brains may operate on similar principles of dynamic fitting - This parallel offers new perspectives on artificial and biological intelligence - Could this research lead to more human-like AI? Or help us better understand our own minds? 👉 Implications for AI Development and Cognitive Science The findings of this paper have far-reaching implications: - It proposes using memory as a metric for assessing LLM performance - This could potentially serve as a new benchmark for evaluating and comparing different LLM architectures - The research blurs the lines between artificial and human intelligence, opening up new avenues for interdisciplinary research in AI and neuroscience 👉 Practical Applications and Future Research The insights from this study could impact various fields: - Education: Developing more effective AI-powered tutoring systems - Healthcare: Creating AI assistants with improved context retention for patient care - Personal Assistants: Designing AI with more human-like memory for natural interactions What applications do you envision for AI with more human-like memory capabilities? This research marks a significant step in our understanding of AI cognition. By drawing parallels between LLMs and human memory, it not only advances our knowledge of artificial intelligence but also offers new perspectives on human cognition.

88

6 Comments