Avishalom Shalit

Very interesting details in the Gemma 2 paper now up on arXiV including e.g. ablation results on some of the new techniques introduced in this version of the model. https://lnkd.in/giX7Uwmj

42

I think AI agentic machine translation has huge potential for improving over traditional neural machine translation, and am releasing as open-source a demonstration I'd been playing with as a fun weekend project. Using an agentic workflow, this demonstration (i) Prompts an LLM to translate from one language to another, (ii) Reflects on the translation to come up with constructive suggestions, (iii) Uses the suggestions to refine the translation. In our limited testing, this is sometimes competitive with, and sometimes worse than, leading commercial providers. But it gives a highly steerable translation system where by simply changing the prompt, you can specify the tone (formal/informal), regional variation (do you want Spanish as spoken in Spain or as spoken in Latin America?), and ensure consistent translation of terms (by providing a glossary). This is not mature software. But I hope the open-source community can make agentic translation work much better. Given how a simple reflection workflow already gives decent results, I think there's significant headroom to make agentic translation much better. Releasing an early software prototype like this is something new I decided to try to see if it is helpful to the developer community. I'd love any feedback on this. Thanks to Joaquin Dominguez, Nedelina Teneva, PhD and John Santerre PhD for help with this. https://lnkd.in/gjGANH6H

5,303

163 Comments

Responsible and accountable AI models are essential as generative AI continues to revolutionize industries. The rise of this technology, however, presents challenges in understanding how these models make decisions. Mechanistic interpretability addresses this by going beyond traditional methods that track statistical relationships. A recent research work by a team from #Anthropic has involved visualizing which neurons activate in response to specific prompts, advancing monosemanticity—ensuring each neuron’s activation correlates to a single, clear function. This step is crucial for fostering more transparent and safer AI systems in various applications. This is a great progress toward interpreting the thinking mechanisms of large language models (LLMs). #GenerativeAI #AI #LLMs #GenerativeAI #Grainger #MachineLearning #DeepLearning #ExplainableAI #DataScience

45

2 Comments

This week, I spoke with Philip Rathle from Neo4j. He just published 💠 The GraphRAG Manifesto: Adding Knowledge to GenAI 💠 where he explains how to use graphs to fine-tune your LLMs and lists the benefits from GraphRAG relative to vector-only RAG: 1. Higher accuracy and more complete answers (runtime / production benefit) 2. Once you’ve created your knowledge graph, then it’s easier to both build5 and subsequently maintain your RAG application (development time benefit) 3. Better explainability, traceability6, and access controls (governance benefit) If you seriously plan to use your data with LLMs, give GraphRAG a try. https://lnkd.in/gdiCq-WU #GartnerDA #GenAI #RAG #GraphRAG

68

4 Comments

LoLCATs: Demystifying Linearized Attention in Large Language Models In their recent blog post "How to Linearize LLMs for Me and You", researchers from Hazy Research introduced LoLCATs, a groundbreaking method to efficiently scale large language models by linearizing attention mechanisms. Large language models (LLMs) like GPT have achieved great success, but they can be extremely computationally expensive, primarily due to the self-attention mechanism, which grows quadratically with the input size. Enter LoLCATs (Learnable Linearized Attention Transformers), a new approach to make LLMs faster and more scalable. Let’s break down LoLCATs, including how they linearize attention, why this is important, and how it works in practice. Self-Attention: Why It’s Powerful but Expensive Traditional self-attention works by comparing every word in a sequence with every other word. In mathematical terms, this creates an N×N attention matrix, where N is the length of the sequence. If you have 1,000 words in a sentence, this requires 1,000,000 comparisons. As sequences get longer, the model has to perform exponentially more work, making it slow and resource-intensive. Linearized Attention: Making Attention Faster LoLCATs linearize attention by approximating this full pairwise comparison process. Instead of calculating all N^2 comparisons, LoLCATs reduce the complexity to O(N). Here’s how: Low-Rank Approximation: Rather than calculating the full attention matrix, LoLCATs decompose it into smaller, more manageable matrices. This low-rank approximation allows the model to compute relationships between tokens without comparing every pair. Factorized Attention: LoLCATs modify the attention mechanism to compute attention scores in two steps: First, it projects the inputs into the query and key spaces using efficient transformations (often kernel-based). Then, instead of computing the full QK^T product, it computes an approximate dot product between queries and keys, reducing the number of operations. The result is that instead of building a full attention map, LoLCATs compress the attention process, maintaining essential relationships without all the heavy computation. Learnable Linear Attention: LoLCATs introduce a learnable linear attention mechanism, allowing the model to improve its approximations over time through training. This makes the system adaptable and able to fine-tune its understanding of relationships between words as the model trains on more data. Why is This Important? By reducing attention complexity to linear time, LoLCATs make it feasible to handle much larger models and longer sequences. For instance, in traditional models, scaling to sequences of 10,000 or 100,000 tokens becomes practically impossible due to resource demands. LoLCATs, by comparison, handle these sequences much more efficiently, reducing both memory usage and processing time. Reference: https://lnkd.in/eJ9XGBmQ

8

2 Comments

Got a chance to write for InfoWorld on why RAG - enabled AI agents are forming the foundation of successful enterprise genAI deployments to automate end to end knowledge workflows. “...the real magic of RAG is in the retrieval model and its upstream components. RAG deployments live and die by the quality of the source content and the retrieval model’s ability to filter the large data source down to useful data points before feeding it to an LLM…” “...a well-designed AI agents approach to the automation of complex knowledge workflows can help mitigate risks with RAG deployments by breaking down large use cases into discrete “jobs to be done…” Auquan is helping teams across private markets, investment banks, insurance firms and other financial services extract meaningful insights from large complex, noisy data quickly and cost effectively, by leveraging RAG and AI agents. If you are similarly looking to buy or build infra to transform troves of valuable unstructured data into timely, actionable intelligence for your firm, give us a shout. Article: https://lnkd.in/gfsis8QB #EnterpriseAI #RAG #AIagents #GenAI #finance

39

I dislike the word "game changer".. but model2vec is a game changer for LLM powered applications needing fast inference operations in Agentic RAG workflows. ⤵ Running a semantic ranking from query... 1.87ms.. less than 2 milliseconds to embed and compare 50 docs to a query. https://lnkd.in/gBSb-5Tx

48

1 Comment

RAG fine-tuning combines code retrieval & model training, addressing outdated knowledge & hallucinations in LLMs. Our experiments show RAG fine-tuned models using the Together AI Fine-tuning API and Morph Labs Code API achieve up to 16% better accuracy, 3.7x faster speed & 150x cost reduction vs Claude 3 Opus, and up to 19% quality improvement, 1.1x faster speed & 37.5x cost reduction vs GPT-4o. #RAG #Finetuning #TogetherAI #MorphLabs

18

Simple messaging, Andrew Ng knows how to simplify complicated topics, here he speaks of AI Agents ... not new but a nice breakdown of agent reasoning design patterns. Consider Reflection, Tool Use, Planning and multi agent collab can help aid solns to be more robust. Some excellent examples that are presented simply. I had the rare pleasure to work with Andrew and many of his graduate students in 2014-2015 around much smaller DNNs for ASR and IRC where we thought a billion params was a lot... how far we have come. https://lnkd.in/gK8DAQMv

25

2 Comments

Data contamination may be the biggest problem in LLM evaluation benchmarks today. Because LLMs have used so much training data, their evaluation sets often include identical or similar data to something they've already seen (i.e. testing on the train set). This completely undermines evaluation results and trust in the model quality. Consider how much GPT-4o performance plummets in the LiveCode code generation benchmark on problems after the cutoff date for its training data (over 20%!) One partial fix is to only ever evaluate LLMs on unseen data collected after the training data cutoff.

12

I am a Unity3D game developer, with a particular passion for AI and multiplayer systems. When ChatGPT emerged, I became fascinated with the idea of creating a hybrid system that combines the strengths of deterministic AI (behavior trees) with stochastic AI (large language models, or LLMs). My initial experiments using APIs (https://lnkd.in/eHvUS3GE) highlighted the cost barriers of adapting such technologies for game development. Interestingly, in those early attempts, character personalities were generated by ChatGPT based on five psychological archetypes, along with details like backstory and profession. With newer models, I’ve tested all affordable options, and so far, the best-performing (price, quality) model has been GPT-4o-mini, delivering responses in just 0.3 seconds. Cost efficiency is critical because, in behavior tree-based AI, decisions are made every frame or server tick (especially in multiplayer games). On average, this happens 10–15 times per second in games like Minecraft. So, to create a truly responsive, "living" AI, the server must provide responses within 0.06 seconds. I was very surprised to learn that the delivery of a live interview is much better than the structured model, indicating personality.

1

1 Comment

Researchers from SambaNova Systems will be at #EMNLP Conference this week. We are presenting 2 papers, 🎉🎉1 of which has won the best paper award 🎉🎉 at Custom4NLP Workshop. 1. SambaLingo: Teaching Large Language Models New Languages - We dive deeper into how we train our Sambalingo series of multilingual models. ✅ This series of models has been downloaded from that 🌟 100K times🌟 and has inspired development of popular models in various geographical locations. ✅ We dive into how to adapt monolingual models to new languages during pre-training, instruction tuning and alignment, understanding the trade offs with respect to tokenizers and translated data ✅ Paper link - https://lnkd.in/gWpfZUrJ 2. Constructing Domain-Specific Evaluation Sets for LLM-as-a-judge - We dive deeper into how to construct benchmarks for domain specific tasks and datasets ✅ The basic idea of the paper is to introduce a data pipeline to generate domain-specific evaluations for LLM-as-a-Judge, tailored for your (or your customer's) usecase! ✅ There are also more interesting experiments like comparing different kinds of judges such as closed source vs open source judges, etc. ✅ Paper Link - https://lnkd.in/gJBvjFGJ Work done by Jonathan Li Ravi Raju Bo Li Swayambhoo Jain Changran Hu Yun Du Qiantong Xu Pian Pawakapan Leon Zhang Zoltan Csaki Reach out to Ravi Raju Bo Li and Zoltan Csaki who are attending the conference if you want to chat about #LLM #Inference, Fast Tokens, the papers above or talk about opportunities at SambaNova Systems #GenerativeAI

78

2 Comments

I just published a blog on how to fine tune LLMs and create custom datasets. In this blog, I cover: - What fine-tuning is and its benefits - When to fine-tune a model - Methods for fine-tuning, including full fine-tuning and parameter-efficient fine-tuning - How to prepare your custom dataset Check it out here: https://lnkd.in/gDcrrBGP

15

1 Comment

NVIDIA just released an open source model that can beat GPT-4o and Claude 3.5 Sonnet👇🏼 - NVIDIA released 70B model for commercial use - Links to try it for free: 1. https://lnkd.in/efXiYnGj 2. https://lnkd.in/eAZCRi7d - NVIDIA wants to make the open source models so good that all the for-profits have to order more chips to train increasingly complex models to distinguish their models to justify charging for access - a Reddit user Thanks Rod Askarov for the insightful post! #ai #chatgpt #aitools #openai

7

3 Comments

AI Capabilities Plateauing? Is "Bigger is Better" Losing Its Edge? A recent article by Reuters highlights these challenges and the industry's pivot toward new paradigms, such as reasoning and inference-time computation. The focus is shifting from brute force scaling to models that innovate and think independently. For years, scaling up pre-training—feeding vast amounts of data into ever-larger models—has been the cornerstone of AI advancements. But according to Ilya Sutskever, co-founder of OpenAI and Safe Superintelligence (SSI), we may be hitting a plateau. Recent findings suggest that simply increasing model size is yielding diminishing returns. Yann LeCun puts it succinctly: "Unless the learning system can 'invent' solutions and answers to new inputs by itself (through an ability to reason) instead of merely receiving and adapting previously-learned answers, the plateau will be with us." Implications for the Industry - Are we ready for AI systems that think and reason rather than memorize and adapt? - How do we redesign learning paradigms to enable true reasoning and problem-solving? This isn’t just a technical shift—it’s a philosophical one. AI’s evolution depends on its ability to move from scaling to reasoning. Reuters Article: https://lnkd.in/e2uyZVVW #AI #MachineLearning #ScalingLaws #Innovation

8

3 Comments

Langchain via LangGraph is democratizing agentic, self reflection and multi agent workflows on steroids

8

This is a great first step in understanding how scaling dynamics extend beyond language to other types of sequential data. Time is a universal axis, understanding how models learn optimally over data collected over time is a crucial step in the broader AI research agenda.

26

Better & Faster Large Language Models via Multi-token Prediction Large language models such as #GPT and #Llama are trained with a next-token prediction loss. In this work, we suggest that training language models to predict multiple future tokens at once results in higher sample efficiency. More specifically, at each position in the training corpus, we ask the model to predict the following n tokens using n independent output heads, operating on top of a shared model trunk. https://lnkd.in/dWUQsyk6

3

1 Comment