Weihua Zhu

What’s the Real Context Size of Your Long-Context LLMs? Picking up from my last post (https://lnkd.in/gYtgs_DT), where I talked about the pros and cons of long-context LLMs in Retrieval-Augmented Generation. This NVIDIA research takes a closer look at how these models perform at their limits, with insightful findings that reveal some non-obvious challenges. The authors tested 17 language models, including well-known ones like Gemini 1.5, GPT-4, and Llama 3.1. Turns out, despite many models claim to handle huge contexts, very few actually keep up their performance as context length grows. This might come as a surprise if you're familiar with traditional benchmarks like "needle-in-a-haystack" and passkey retrieval, where most models achieve near-perfect accuracy. While those tests are useful, the new RULER benchmark pushes LLMs with more complex tasks, such as multi-hop tracing and aggregating information across long contexts, ultimately revealing their true limitations and areas for improvement. As language models grow in size and context capacity, it's easy to get excited about their new capabilities and forget the trade-offs. Just because a model can handle massive contexts doesn’t mean more is always better — often, keeping it small and simple still wins the day. I'll drop the link to the paper in the comments. #MachineLearning #ArtificialIntelligence #AI #LLM #GenAI #NeuralNetworks

57

3 Comments

Everyone knows Moore’s law, the idea that CPU performance doubles roughly every two years. But remarkably, driven by GPU performance and AI model breakthroughs, we’ve seen a very different law emerge with respect to the rate at which AI model performance improves — in many cases improving by orders of magnitude in the past decade. This introduces massive implications in a world of AI Agents. For the past five plus decades, in any domain that computers touch, we’ve generally seen massive efficiency gains that bring more tech to more people. Yet the same has not been possible for essentially any non-digital services. Historically, anytime you want to solve non-computerized business problems (i.e. most knowledge work), we have seen the cost to solve a problem — say from healthcare services to legal work — tend to only go up over time. Due to a mix of inflation, specialization in certain fields, regulation, decreased competition in markets, or other added costs, we’ve seen the price of most services generally monotonically increase over time. This also puts a burden on driving economic activity in general, especially when creating new ideas from scratch, or starting and running a new business. Yet, in a world of AI Agents, anyone — from a small business or large enterprise — has access to automated work that can let them address problems with variable capacity. When you decrease the cost of entry for getting knowledge work done, you dramatically increase the use cases for that work. That new sales program can be tested instantly instead of waiting weeks or months or never happening at all; you can test a broader surface area of your product for bugs or security issues; legal work and reviews becomes affordable on the seemingly insignificant transactions that otherwise produce risk; or you can launch a new marketing campaign in markets you didn’t otherwise serve. Even in a world of AI costs remaining stable this is a super compelling proposition, but in a world of constantly improving technology the implications are vast. Just as Moore’s law steadily lowered the cost of software, increased its sophistication, and made it more ubiquitous, we can imagine a world where knowledge work regularly gets more affordable, sophisticated, and ubiquitous at a constant rate. This means AI Agents present the ability for any unit of work to continuously drop overtime, enabling us to solve greater and greater problems for a wider set of people and businesses. Of course, a large portion of these efficiency gains will quickly get “eaten up” by solving increasingly more sophisticated problems, but those problems will be solved at the cost of a previously simpler problem. The implications of this are enormously interesting, and it’s going to create incredible opportunities all around.

151

13 Comments

Sharing my thoughts on How Waymo (formerly known as the Google Self-Driving Car Project) is Pioneering Autonomous Driving with AI. Waymo has become a trailblazer in self-driving technology, and at the core of it all is AI. Leveraging advanced machine learning and deep neural networks, Waymo’s AI-powered system continuously improves through real-world data, making autonomous vehicles smarter, safer, and more reliable. Perception: AI allows Waymo’s cars to "see" and understand their surroundings by processing data from sensors like cameras, LIDAR, and radar. This helps detect pedestrians, other vehicles, and objects in real time. Prediction: The AI system predicts the behavior of everyone around the vehicle—whether it’s cars changing lanes or pedestrians crossing the street—ensuring smooth and safe driving decisions. Planning & Control: AI makes split-second decisions about accelerating, braking, and steering based on the analysis of complex traffic patterns, enabling Waymo vehicles to safely navigate roads. With millions of autonomous miles driven, Waymo’s AI is constantly learning and improving, helping create the future of safer and smarter transportation. The road ahead is AI-driven! www.rxsmeta.com #AI #AutonomousVehicles #Waymo #SelfDrivingCars #MachineLearning #SmartMobility #TransportationTech #FutureOfTransport #AIinAction #Innovation

2

1 Comment

We are excited to back Morphing Machines. As Dr. Ranjani, CTO and co-founder says: “For the first time in India and globally, we have a processor that is truly runtime reconfigurable, catering to emerging workloads.” While Nvidia’s growth and high valuation are driven by robust demand for “infra-compute” in high-performance GPUs, spurred by Gen AI, there is a distinct need for innovative solutions tailored to a new wave of reactive applications. Enter Morphing Machines. It addresses this demand with its unique product designed for next-gen telecom, autonomous vehicles, collaborative robots, avionics, and online data analytics. This product is specifically engineered to handle heterogeneous workloads and diverse tasks with varying performance demands simultaneously, ensuring a high level of scalability. As we move from an institutionally-backed research team at IISC to productization, Golden Sparrow is collaborating with Speciale Invest and others to support Deepak Shapeti, who joined as CEO three years ago after earning his Masters at Stanford. He brings not only technical expertise but also the business acumen necessary to guide the product to commercial success. Venkat Raju Afreed Faizan Anand Rao Ritik Singh Michael Marmor

68

5 Comments

A new tool for evaluating LLM. This could be useful for enterprises.

7

Chip Huyen on Twitter today below. I 100% agree. Metric hygiene is system hygiene. LLM-Generated metrics require a higher standard of care than they're currently receiving, and much more scrutiny. Mind the compasses. 🐦  A big issue I see with AI systems is that people aren't spending enough time evaluating their evaluation pipeline. 1. Most teams use more than one metrics (3-7 metrics in general) to evaluate their applications, which is a good practice. However, very few are measuring the correlation between these metrics.If two metrics are perfectly correlated, you probably don't need both of them. If two metrics strongly disagree with each other, either this reveals something important about your system, or your metrics just aren't trustworthy.2. Many (I estimate 60 - 70%?) use AI to evaluate AI responses, with common criteria being conciseness, relevance, coherence, faithfulness, etc. I find AI-as-a-judge very promising, and expect to see more of this approach in the future.AI-as-a-judge scores aren’t deterministic the way classification F1 scores or accuracy are. They depend on the judge's model, the judge's prompt, and the use case. Many AI judges are good, but many are bad.Yet, very few are doing experiments to evaluate their AI judges. Are good responses given better scores? How reproducible the scores are -- if you ask the judge twice, do you get the same score? Is the judge's prompt optimal? Some aren’t even aware of the prompts their applications are using, because they use prompts created by eval tools or by other teams.Also fun fact I learned from a (small) poll yesterday: some teams are spending more money on evaluating responses than on generating responses 🤯 https://lnkd.in/gYnTRZW3

2

Optillm: An optimizing inference proxy for LLMs Optillm is an optimizing inference proxy that has over a dozen techniques that aim to improve the accuracy of the responses using test-time compute. Over the last couple of months we have set several SOTA results using smaller and less capable models like gpt-4o-mini. Recently, we have added support for plugins that enable capabilities like memory, privacy and code execution to optillm. Plugins are just python scripts that you can also write yourself, optillm would then load them at start from the directory. You can now also combine the plugins and techniques using & and | operators. E.g. We recently evaluated the new FRAMES benchmark from Google. Using a combination of plugins and techniques (we used readurls&memory-gpt-4o-mini) we were able to get 65.7% accuracy on the benchmark which is very close to what Google reported in their paper with Gemini Flash 1.5 (66.5) which has a context length that is almost 10 times that of gpt-4o-mini. Do check out Optillm at https://lnkd.in/gN6_kNky

21

1 Comment

Polaris 2.0 from Hippocratic AI is here. Safer than ever. 3T+ parameter constellation architecture with voice latency equal to Polaris 1.0 at 1T+ parameters. Read a summary of the changes from our chief scientist Subhabrata (Subho) Mukherjee's blog: https://lnkd.in/gXweaWe5

278

1 Comment

Trim chips, not precision! :). Check out llama-8b running on 8 sockets at 430 tokens/second and 16 bits! The power of dataflow architecture is on full display here. And we have just scratched the surface! A lot more to come. It was also very exciting to see that this update was covered in Andrew Ng's news letter! If you want to work on applications that can be enabled by these new capabilities or are want to work on LLMs, Multimodality, model merging and other techniques, or the infrastructure that supports these capabilities, we are hiring! #generativeAI #LLM

33

Seeking a full time onsite hybrid AI Architect Engineer Manager for leading RAG and LLM Generative AI role to relocate and lead RLHF, prompt engineering for zero shot learning, one shot learning, few short learning. Engineer Architect or Manage and lead teams on Fine Tuning LLM models on PEFT, RLFL, Prompt Tuning context window, and to work on Reward model using LoRA, and to work on agent to instruct LLM on Context window and lead AI Generative LLM RAG performance for scalability. Engineer Architect or Manage and lead Generative AI LLM Lang Chain Use cases, and work on Hugging faces for encoded/weights. Engineer Architect or Manage and contribute to building RAG Retrieval Augmented Generation to retrieve external library data for tailored to specific models domain in prompt analysis. Additional talents that I have which include People first partnerships, deliver on AI architect accountability and ensure focus on diversity inclusion for delivery business value. Manage stakeholders data projects to improve business revenue. People talents that I offer are people loving manager who is humble and offers critical creative innovative AI thinking with the team. Engineer Architect or Manager the architecture for AI Lang Chain Generative AI Use Cases that need LLM text words. Contribute as Engineer Lead on improved production pipeline supply chain signal processing and be the Engineer Architect for large language text analysis (LLM). Research for teams can be more productive in using Lang Smith for LLM large language models. Assist on curating sandbox to develop bidirectional decoder/encoder transformer (BERT) masking language next sentence (MLM). Lead again on OpenAI GPT for optimized performance to build chat what-if analysis for the business. Manage Engineer to lead scaling the team on development with performing LLM Chain and predictions models and build predict messages that pull from the Hugging Face Hub objects and prompt engineering with prompt templates and human message responses.

3

“Models like Llama 3, Mixtral are “open-weight models”, not “open-source models”. For a model to be open-source, we need full information about the processing code and data.”

4

1 Comment

📝 Exciting news! My latest blog post on Medium is now live! In this multi-series blog, I explore anomaly detection in any system. I share my experiences in designing and implementing an end-to-end solution for anomaly detection patterns in user access patterns. This series will cover a variety of topics, including understanding application events, optimizing data in Snowflake, developing machine learning models, and integrating with external applications. In the first blog post, I dive into everything about events and how they should be handled in a distributed system. Check out the full article here to learn more: https://lnkd.in/g8mjARdd I would love to hear your thoughts and insights in the comments section! Let's start a conversation. #events #eventing #outboxpattern #anomalydetection #machinelearning #dataoptimization #AWS #Snowflake

63

5 Comments

Insightful Paper on Scaling law for Precision TL;DR The more tokens you train on, the more precision you need for inference. The paper studies the limits of quantization that has implications on compute resources, model size decisions. - Training in lower precision reduces a model's effective parameter count. - Post-training quantization degrades performance more for models trained on larger datasets. - Paper also suggested a scaling laws to predict the loss of a model with different parts in different precisions, and suggest that training larger models in lower precision may be compute optimal. https://lnkd.in/gk8yZx_p #quantization #LLMs #AI #inference

19

Hey, I'd like to give a shout-out to ClearML for helping us at Vimeo develop AI-based systems. We have been happy customers for a few years now and I'd like to share a bit about how we use ClearML. We use ClearML Datasets to cache a dataset of video transcripts, and run testing loading directly from ClearML Datasets. This allows us to simultaneously speed up the data loading and ensure consistency. Moreover we use ClearML to save our benchmark annotations in one place and track our system performance on the benchmark every-time we run an experiment or change our prompts. ClearML allows us to see the different parameters and prompts that were used for each experiment and to monitor improvements or regressions in our performance. We also use ClearML to run large-scale tests and help with statistical evaluation of our methods. For example, we developed a RAG (Retrieval Augmented Generation) Q&A system and wanted to verify that the LLM will not answer certain questions or user queries that are outside the scope of the video. We used ClearML to collect and analyze the RAG responses on many videos for predefined user queries that were outside the scope of the videos and got good visibility into the performance of the system. Also, the comparison feature on ClearML is great for tracking the improvement of our metrics along the progressing versions of our systems.

32

1 Comment

Struggling to make LLMs generalize to out-of-distribution data? Your frustration might be alleviated by an ICLR 2024 paper, which rigorously proved that learning an approximate causal model is a necessity for OOD generalization. The result has far broader implications, as detailed in Section 4. One thing I'm speculating is that learning a causal model is the only way to be sample efficient and is the fundamental difference between LLMs and human brains. #artificialintelligence #machinelearning #deeplearning https://lnkd.in/eUPFa3fw

57