Alhussein Fawzi

In-Context Learning at Extreme Scales: My Thoughts As large language models (LLMs) continue to expand their context capacities, the domain of in-context learning (ICL) at extreme scales has piqued my interest. A recent paper by researchers at Carnegie Mellon University and Tel Aviv University provides a detailed exploration into this area, unveiling both impressive capabilities and notable challenges. 🔍 Key Insights: Performance Improvement: The study demonstrates that ICL can improve significantly by integrating hundreds to thousands of examples into the context window—far beyond traditional few-shot settings. This is particularly impactful in complex tasks like 77-way banking intent classification, where long-context ICL matches or surpasses traditional finetuning. Input Sensitivity: Long-context ICL shows reduced sensitivity to input shuffling compared to shorter contexts. However, organizing examples by label dramatically reduces performance, emphasizing the need for diverse context configurations. Source of Gains: Performance gains primarily arise from accessing more relevant examples, rather than aggregating knowledge across many examples during encoding. 📉 Challenges & Drawbacks: Sensitivity to Example Order: Despite reduced sensitivity, the performance still varies with input order, indicating potential instability in practical scenarios. Performance Saturation: The models often do not fully leverage their extended context capacities, showing performance saturation before reaching the maximum context length. Computational Cost: The increased computational cost associated with using large context windows could limit practical deployment, particularly in resource-constrained environments. Efficiency Issues: There is a noted inefficiency in how models utilize the encoded information within large datasets, suggesting that current architectures may need refinement to better use available data. 🚀 Future Directions: The paper suggests several exciting avenues for future research, including optimizing the trade-offs between computational costs and performance, exploring model adaptation strategies, and refining ICL approaches for specific applications or datasets. As we push the boundaries of what LLMs can achieve, effectively leveraging their full context could unlock new capabilities. This area is ripe for further exploration and could lead to major advancements in how we deploy AI. For those at the cutting edge of language model development, I highly recommend this paper for its thorough analysis and forward-looking perspective. Check out the full work at https://lnkd.in/d7AmGh4y. What are your thoughts on the future of in-context learning at such scales and the challenges discussed? #AI #MachineLearning #DeepLearning #LanguageModels #InContextLearning

17

The Transformer Series: This video explains the concept of positional encoding in the transformer architecture. In our previous videos, we looked at Skip connections, scaled dot product attention and multi-head attention. Now, let’s shift our focus to another important aspect of the transformer architecture: Positional Encoding. Positional encoding is crucial for transformers to understand the order of input data when looking at sequential data, such as the usual english sentences where the order of the words is important. Positional encoding adds unique markers to each input token to provide context of its position in the sequence. This method allows the model to recognize patterns and relationships in data that depend on the sequence order—something essential in tasks like language understanding or time series analysis. Take a look at this video to discover how positional encoding enables transformers to map out complex underlying patterns in sequence data. https://lnkd.in/g4bi4_A3 #transformers #deeplearning #ai #ml #nlp

59

Implementing Reinforcement Learning (RL) in the context of Large Language Models (LLMs) posed challenges, particularly in designing reward engineering and penalization components. In traditional RL, reward engineering is straightforward, often defined through a scalar reward function. However, implementing the same concept in RL with LLMs required a critic model that evaluates the questions and answers, critiquing them in a manner that simulates the reward function. A recent research paper on Natural Language Reinforcement Learning (NLRL) provides an intriguing perspective on the reward problem in RL by translating traditional RL concepts into natural language constructs. Here's a summary of how it addresses the reward problem: Language-Based Value Functions: Unlike traditional RL, which uses scalar rewards, NLRL employs language-based value functions. These functions articulate the effectiveness of policies and actions in a more interpretable, natural language format. This approach addresses the need for richer, more informative feedback than simple numerical scores. Iterative Refinement of Language Policies and Critics: NLRL incorporates a structured training pipeline where language-based policies and critics are iteratively refined using textual feedback. This method allows the system to handle the complexities of defining rewards in natural language, improving its understanding and evaluation of actions over time through textual feedback loops. Use of Large Language Models (LLMs): The framework leverages the advanced capabilities of LLMs to interpret, generate, and evaluate complex language-based information. This is essential for implementing language-based value functions and policies, enabling the system to generate and refine responses that align closely with desired outcomes. Distilling Enhanced Actions and Evaluations: NLRL employs a process where enhanced evaluations from the language-based value function approximator and improved actions from the language policy improvement operator are distilled. This allows for continuous refinement and learning. Compatibility with Existing LLMs: The framework is designed to be compatible with current LLMs, enabling it to leverage pre-existing language models to process and generate linguistic data effectively. This compatibility is crucial for translating the traditional reward engineering process into a language-based framework. The NLRL framework's introduction of natural language as a medium for representing and evaluating policies and rewards offers a novel solution to the reward problem in RL. By shifting from numerical reward systems to a more flexible, expressive, and intuitive system based on language, NLRL enhances interpretability and adaptability, paving the way for more diverse RL applications. Innovation Hacks AI Inc. #llm #refincorcementlearning #ai

4

🤖 Learn how to build a state-of-the-art chatbot application optimized for mobile processors, using tools like the OpenVINO™ Toolkit and PyTorch with Paula Ramos, PhD during her session at #AllThingsOpen happening today at 1:45pm.

42

Quantized Side Tuning: Fast and Memory-Efficient Tuning of Quantized Large Language Models: There are two existing approaches to fine-tuning: 1. Parameter efficient fine-tuning (PEFT) (LoRa, QLoRa): update a small number of parameters; the rest will freeze. 2. Reduce the memory footprint during the training phase of fine-tuning. However, PEFT needs caching intermediate activations during forward processing, which takes the same training time as full fine-tuning and can’t reduce the memory footprint. Existing methods require a considerable amount of memory for fine-tuning. Quantized side tuning (QST) quantized LLM weights into 4-bit to reduce the memory footprint. Other models need backpropagation where QST utilizes the hidden state of LLM as a task-specific using a side network separated from LLM. QST uses several low-rank adaptors, and gradient-free downsample modules to reduce trainable parameters. https://lnkd.in/gjpK2s7w

7

Our paper, titled 'Stereographic Spherical Sliced Wasserstein Distances,' just got accepted to #ICML with stellar reviews. In this work, we propose a computationally efficient metric for comparing spherical distributions and demonstrate its numerous applications in ML problems. Congratulations to Huy Tran and the hardworking team behind this paper at Vanderbilt University Department of Computer Science ! Link to preprint: https://lnkd.in/gaSu4uJE

108

8 Comments

Pretty cool paper by Tamer Abuelsaad, Deepak Akkil Ravi Kokku and team on their implementation of Agent-E [1,2] - a multiagent workflow (built with #autogen) for accomplishing web based tasks (driving web interfaces). Paper: https://lnkd.in/gNk-nUAY The paper includes lots of nuggets that help us improve on our collective understanding of how to build interface agents[3], including: - Hierarchical agent architecture with separation of roles e.g, planning, web navigation, etc and the benefits it provides (task verification, error recovery etc) - DOM distillation to reduce noise in web pages. - Results (based on evaluation on the WebVoyaGer benchmark) indicate that a text only representation the use (DOM) does outperform existing multimodal approaches. Great example of where a multi-agent but less powerful setup can result in overall better performance. References [1] Paper on arxiv https://lnkd.in/gNk-nUAY [2] Paper github repo: https://lnkd.in/gaXyuTje [3] Interface Agents - Building Multi-Agent Applications that Act via Controlling Interfaces (Browsers, Apps) https://lnkd.in/eVdFqc_J #autogen #multiagent #multiagentsystems

59

1 Comment

🚀 Speculative Decoding v2.0 🚀 We all want to make LLMs faster! But how!!!🤔 Check out my latest blog, where I describe our latest work on the Mixture of Attentions architecture, a new approach to speculative decoding that speeds up LLM inference by 9.5% and improves token acceptance by 25%! 🔥 Ze Blog: https://lnkd.in/egx5bTXG Check out the paper as well: https://lnkd.in/eYZUbbTe #AI #MachineLearning

15

IndicGenBench Google Research India recently released IndicGenBench, a multilingual benchmark to evaluate generation capabilities of LLMs on 29 Indic languages spanning 13 writing scripts and 4 language families. Extended the datasets in Cross-lingual Summarization, Machine Translation, Multi-lingual Question Answering, and Cross-lingual Question Answering, the team has collected human translations for English examples into target Indic languages, thereby extending the scope and applicability of evaluation metrics in this domain. One of the key insights from their study is the analysis of token fertility across all Indic languages within IndicGenBench. Token fertility, representing the average number of sub-words that a word is broken down into by the tokenizer, varies significantly across languages Some languages have simple breakdowns, while others are more complex. Now, why does this matter? Well, it affects how well the language models work. Languages with more complex breakdowns might struggle because they can't use as many examples to learn from. They found that languages with simpler breakdowns can use more examples effectively compared to those with complex breakdowns. #LLMs #GenAI #IndicGenBench #AI #IndicLanguages #IndicDatasets #multilingual

15

1 Comment

Large time-series models (LTMs) enjoy similar power-law scaling behaviour to LLMs. We just put out a paper (https://lnkd.in/givY528D) establishing power-law like scaling-laws for large time-series models as a function of data, compute, and model size. Similar scaling-laws for LLMs (from the landmark Kaplan et al. paper https://lnkd.in/g9KHYN9u) have provided key guidance in allocating enormous resources for predictable - and eventually breakthrough - performance gains. The demonstration of similarly favourable scaling behaviour for large time-series models provides both a motivation and guide in the pursuit of foundation models for time-series forecasting. Foundation models for time-series are coming (with enough data and compute). Thanks to Thomas Edwards, James Alvey, Benjamin Wandelt and Nam Nguyen for the hard work!

48

1 Comment

This new article discusses our efforts in delivering tensor parallelism, weight-only quantization, vLLM integration, and the RBLN model zoo. It is definitely worth reading! 👉 https://lnkd.in/gRnG8_pR #TensorParallelism #WeightOnlyQuantization #vLLM #ModelZoo

84

Mamba S5 - Selective Structured State Space Models! This blog took a while to complete! Foundation models often use the Transformer architecture, which faces inefficiencies with long sequences. Mamba AI improves this by enhancing state space models (SSMs) to dynamically respond to inputs, allowing selective information processing. This results in better efficiency and performance without traditional convolutions, significantly outperforming Transformers in speed and scalability across multiple modalities. #ArtificialIntelligence #MachineLearning #Math

1

Excited to share the work that Chris Glaze Changho Shin Frederic Sala and I did on long context LLM evaluations. While there’s plenty of problems with LLM evals, a recurring one we’ve seen at Snorkel, working with enterprise clients, is having to rely on far too simple/toy benchmarks that do not relate to business problems. It is why we developed SWiM, a framework that lets users test long context model capabilities directly on their own data and tasks. Read the paper here: https://lnkd.in/g3uSuKcW

31

How can LLMs aid in the process of building Simulation models? Some weeks ago, my (first) Journal article was published! There, we examined the potential of prompting ChatGPT, to generate functional Simulation model code from a prose-based narrative. Please check it out! #LLMs #AI #GenAI

8

1 Comment

Karpathy recalls Yann's "cake" after almost a decade. It is amazing how top researchers (and their intuitions) fall prey to well-hyped "local minima" of ideas, over and over again. and then there are some who don't. While he downplays RLHF to be the petite cherry on the cake, the dust hasn't settled on that yet. In spite of DPO, SPO, and what not, researchers keep writing paper saying that the cherry is the crowning glory of the cake. --- Get the latest AI news and updates at offnote.substack.com.

45

What a great talk from Swapnil Sayan Saha, on Physics-Aware Tiny Machine Learning! https://lnkd.in/g9g4YWs4 There's exciting times ahead as Physics-aware TinyML is becoming increasingly important in merging the power of machine learning with operating in the physical world. By integrating real-world physics models into ultra-low-power devices, this approach dramatically enhances the accuracy and efficiency of TinyML applications. From improving energy efficiency in smart sensors to enabling more accurate anomaly detection in industrial applications, this approach to machine learning at the edge helps machines better understand their environment - without sacrificing power or performance. As we explore the possibilities, this combination of physics and Edge AI opens up new opportunities in fields like healthcare, environmental monitoring, and autonomous systems. It's a huge leap toward making AI more sustainable and impactful, and allows us to build more useful AI-enabled systems that operate in (and on!) the physical world. #TinyML #EdgeAI #PhysicsAwareAI #AIInnovation #Sustainability #MachineLearning

4

We are still hardly scratching the surface of what is possible with LLMs.

30

2 Comments

My team is looking for new Research Scientists to join in Meta's London office. Suitable for recent PhD graduates and postdocs. For any questions, please reach out to Daniel or to me. Some recent works from our team: • 𝗜𝗻𝗳𝗼𝗦𝗵𝗮𝗽 (NeurIPS 2023), a SHAP extension to explain uncertainty estimates: https://lnkd.in/d4DPic5j • 𝗧𝗖𝗘 (UAI 2023), a metric to quantify probability calibration error that is suitable for extreme class imbalance: https://lnkd.in/dHbbQ6Nf • 𝗨𝗖𝗕-𝗘𝗨 (Data Mining & Knowledge Discovery journal 2024), an Active Learning method that accounts for non-response in label requests: https://lnkd.in/ejKSuNq9 • 𝗔𝗱𝗮𝗽𝘁𝗶𝘃𝗲-𝗪𝗲𝗶𝗴𝗵𝘁 𝗦𝗮𝗺𝗽𝗹𝗶𝗻𝗴 (under review), an Active Learning method with provable convergence guarantees that can handle industry scale data volume: https://lnkd.in/eQYnD4HB • 𝗢𝗻𝗹𝗶𝗻𝗲 𝗹𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗯𝗲𝘀𝘁 𝗮𝗰𝘁𝗶𝗼𝗻 𝗾𝘂𝗲𝗿𝗶𝗲𝘀 (under review), a theoretical analysis of an online learning model that is applicable to industry scale content moderation systems: https://lnkd.in/eTCKAXFt

74