ZHAN CHEN

We’ve run almost a billion inference requests at Beam Here are a few things we’ve learned about real-time inference APIs 1. For small models, you can often run multiple workers in parallel on the same GPU and increase throughput 2. Weights should be cached on disk or in distributed storage. You don't want to be downloading large files from cloud storage during an inference request 3. Cross-region latency matters. Regional caching is hard and requires a lot of planning 4. You can't use the same autoscaling strategy for every model. You have to see how your workload performs in the real world and choose a strategy that makes sense When you run APIs powered by large ML models, performance isn’t guaranteed. What other strategies have you employed in your ML pipelines to optimize performance?

37

1 Comment

Meet the AI compute engine that solves all sorts of data complexities 🤖 Catch #theCUBE’s top moments from #RaySummit and study up about Anyscale’s newly-introduced AI compute engine called “Ray” from Robert Nishihara, the co-founder. “The complexity of scaling data, new data types, data modalities, scaling compute, and accelerators is skyrocketing. To reign in this complexity, you need an AI compute engine, and that's what Ray is. It solves a handful of core compute-related tasks that's managing the underlying compute resources like auto-scaling, selecting instance types, spot instances, on-demand instances, and handling hardware failures,” Nishihara shares. “Then, there's data management, which is everything from optimizing the data movement through communication frameworks like NI-RDMA, leveraging shared memory, handling large data, so compute resource management, data management, and workload execution. That has to be flexible to support a variety of AI workloads. It has to be fault-tolerant and leverage accelerators,” he furthers. 📺 Watch the full interview! https://lnkd.in/eQjnMEXC #EnterpriseAI #CTOtrends #ScalingAI #CloudComputing #MarketInsights #DataManagement

34

Introducing RouteLLM: a sophisticated routing framework developed with Berkeley-LMSys in collaboration with Anyscale. RouteLLM optimizes query handling by dynamically selecting between high-performance proprietary LLMs and cost-effective open-source models, cutting costs by over 2x without sacrificing quality. Using human preference data and LLM-as-a-judge for data augmentation, our routers evaluate query complexity to choose the appropriate model. Rigorous testing on benchmarks like MMLU and GSM8K confirms our cost-efficient, high-quality performance. Explore our open-source code, models, and preference data on GitHub: https://lnkd.in/gJyijZRx, and try our online demo: https://lnkd.in/gME-GC2a Explore More about RouteLLM and learn how it can revolutionize your LLM applications: Read our Blog here: https://lnkd.in/gfPD-u-y LLMsys Blog here: https://lnkd.in/ga_MgERE Full research paper here: https://lnkd.in/gqRy7Pjy

81

1 Comment

🤖 Learn how to build a state-of-the-art chatbot application optimized for mobile processors, using tools like the OpenVINO™ Toolkit and PyTorch with Paula Ramos, PhD during her session at #AllThingsOpen happening today at 1:45pm.

42

Interested in fine-tuning vision language models and their real-world applications? Check out our latest work with fine-tuned LLaVA, where Yanwei (Wayne) Zhang and our Data team achieved better-than-human accuracy in some e-commerce applications. This approach enhances model accuracy, streamlines user experience, and boosts operational efficiency through advanced automation. They also explore extending this to a general instruction-following framework for tasks like recommending product titles, descriptions, and extracting attributes such as color, materials, weights, and dimensions. https://lnkd.in/gqSAz_4J

12

1 Comment

🚀 Looking to streamline fine-tuning for models like Llama 3, Mistral, and Mixtral? Our latest blog shows how Anyscale simplifies LLMs fine-tuning with Ray: 🔑 Key takeaways: • Scalable fine-tuning with LLM-Forge • Model Deployment with RayLLM Read more: https://lnkd.in/gQwD4A2U

34

The parameters of compiler optimizations interact in complex and non-trivial ways, and finding optimal combinations remains an open research challenge. There are several approaches to discovering effective combinations of these optimization parameters. Recently, Michael Canesche (Cadence Design Systems) and Gaurav Verma (Stony Brook University) have had promising results using auto-tuning with coordinate descent. To visualize this process, imagine that each optimization represents a 'dimension' in a search space. The combination of all optimizations and their respective parameters forms a vector in this space. Now, consider that each point in this space is mapped to a real number representing the performance of a kernel (assuming a fixed input). Coordinate descent is a search technique that assumes the performance curve is convex. Based on this assumption, it aims to find the optimal point by varying one optimization parameter at a time. Explore as a Storm, Exploit as a Raindrop: On the Benefit of Fine-Tuning Kernel Schedulers with Coordinate Descent: https://lnkd.in/dgxxfdYR The Droplet Search Algorithm for Kernel Scheduling: https://lnkd.in/dBfQid9z

77

If you're running on constrained hardware and/or with no GPU, LLaMA 3.2 1B models with 4-bit quantization work surprisingly well! The model is only 800MB. Here's an easy-to-try example with txtai! Example: https://lnkd.in/etd5w-9Q

8

2 Comments

New on Runtime: Anyscale is built around Ray, an open-source project that was designed to help AI workloads scale. But in recent years, commercial pressures have forced several companies with similar open-source origin stories to put restrictions on their projects to ward off competition. How will Anyscale resist the same pressure as it grows? As Robert Nishihara put it, "The value of our platform, of our product, can't just be managing the open-source project. That's not enough." Part one of our interview with Robert and new Anyscale CEO Keerti Melkote ran in last week's Runtime newsletter, and in this Q&A, we focus on the opportunities and pitfalls of building an infrastructure business around an open-source project.

11

1 Comment

Optillm: An optimizing inference proxy for LLMs Optillm is an optimizing inference proxy that has over a dozen techniques that aim to improve the accuracy of the responses using test-time compute. Over the last couple of months we have set several SOTA results using smaller and less capable models like gpt-4o-mini. Recently, we have added support for plugins that enable capabilities like memory, privacy and code execution to optillm. Plugins are just python scripts that you can also write yourself, optillm would then load them at start from the directory. You can now also combine the plugins and techniques using & and | operators. E.g. We recently evaluated the new FRAMES benchmark from Google. Using a combination of plugins and techniques (we used readurls&memory-gpt-4o-mini) we were able to get 65.7% accuracy on the benchmark which is very close to what Google reported in their paper with Gemini Flash 1.5 (66.5) which has a context length that is almost 10 times that of gpt-4o-mini. Do check out Optillm at https://lnkd.in/gN6_kNky

21

1 Comment

It’s day 4 of our 11 Days of Inference Acceleration Techniques. Today, we’re moving on to runtime level optimization best practices. 𝐅𝐨𝐮𝐫𝐭𝐡 𝐭𝐢𝐩: 𝐓𝐚𝐤𝐞 𝐚𝐝𝐯𝐚𝐧𝐭𝐚𝐠𝐞 𝐨𝐟 𝐠𝐫𝐚𝐩𝐡 𝐜𝐨𝐦𝐩𝐢𝐥𝐚𝐭𝐢𝐨𝐧 📈 Graph compilers such as TVM, Tensor-RT, and OpenVino work by getting a computation graph of a specific model and generating an optimized code adjusted for the target hardware. Graph compilation can optimize the graph structure by merging redundant operations, performing kernel auto-tuning, enhancing memory reuse, preventing cache misses, and more. But few things to be aware of: 📝 Not all models compile equally. 📝 The impact of compilation on model performance can vary. Hence, make sure to check the architecture's ability to compile early on in the process to avoid wasting time and resources on training a model that can’t be optimized for fast inference. – What’s the #11DaysofInferenceAccelerationTechniques? The Deci team is posting, for 11 days, a series of inference acceleration techniques for deep learning applications. If you’re looking for practical tips and best practices for improving inference, follow Deci AI so you won’t miss an update. #deeplearning #machinelearning #neuralnetworks #computervision

9

Looking to run end-to-end machine learning workloads without external dependencies? Be sure to check out Census on GitHub! This sample code will show you how to use Intel Distribution of Modin and Intel Extension for Scikit-learn to build and run performant #ML workloads. https://intel.ly/3ygmSFr #MachineLearning #Developer #oneAPI

11

Easily generate one billion embeddings in under a day... and with limited budget! How? With Ray Data from Anyscale and Pinecone! Cheng Su and Roy Miara present this can't-miss session during the #DataAISummit tomorrow. Catch it in-person or online.

43

2 Comments

7 ways to slash latency in System Design: When I worked on Meta's distributed data store, managing massive data wasn't the most challenging part – fighting latency was. After years of trial and error, I'm sharing 7 proven best practices for building low-latency systems. 1. Choose the correct architecture Swap slower monolithic architectures for modular, scalable systems that cut latency by operating independently. 2. Optimize data management Choose SQL for structured data or NoSQL for flexibility, and use indexing, sharding, and replication to speed up data retrieval. 3. Improve network design Reduce data travel with CDNs, load balancers, and edge caching. 4. Use efficient communication protocols Use fast protocols like HTTP/2, WebSockets, and gRPC to streamline communication and cut round-trip times. 5. Optimize your code Use efficient algorithms, data structures, parallel processing, and I/O reduction to speed up execution and reduce bottlenecks. 6. Use high-performance hardware Opt for SSDs over HDDs and low-latency cloud services like AWS Global Accelerator for faster performance. 7. Implement smart caching strategies Cache at multiple layers with LRU/LFU eviction and sync updates to avoid stale data. Read the full blog for a closer look at these 7 practices and how to implement them. (Your system—and users—will thank you). https://educat.tv/3B5Dh0N #SystemDesign #SoftwareEngineer

20

Want to write faster, more efficient C++ code? ⚡ Join Wenjin YU's talk for an in-depth session on "Unlocking Performance in C++: From Profiling to Optimization” where you’ll explore essential techniques to identify and tackle bottlenecks before jumping into optimization.  Gain insights into profiling tools and techniques — like code instrumentation and CPU cycle counting — and learn which methods work best for each scenario. 👉 Can’t miss it: https://tiny.pl/f_5bqwmq #codedive #Cpp #PerformanceOptimization 

4

Less than 20 lines of code is all it takes to get started with Llama-3 on Intel Xeon Processors. Get started with the tutorial below. #IntelXeon #LLM #Developer

6

🚀 How much does it costs to Training LLMs and Where Smaller Players Fit In? Training Large Language Models (LLMs) like GPT-4, LLaMA 3, and Gemini is not a means feat. It requires massive amounts of data, compute power, and financial investment. As these models grow, so do the resources needed. Here's a rough breakdown of the costs: 💡 GPT-4: OpenAI spent $100 million+ to train GPT-4, running tens of thousands of GPUs for weeks. The energy used was comparable to powering entire towns. 💡 LLaMA 3 (Meta): Meta trained LLaMA 3 using huge datasets and a distributed compute setup, likely spending similar amounts as GPT-4. 💡 Gemini (Google DeepMind): Google used its global data center infrastructure to train Gemini, a model with hundreds of billions of parameters, likely exceeding $100 million in costs. 🔋So where does these costs come from?: 1. Data: These models consume terabytes of text data, processed and structured before feeding into training pipelines. 2. Compute: Thousands of GPUs or TPUs power high-performance clusters that run continuously for weeks or months. 3. Energy: GPT-3 used 1.3 GWh of electricity during training, which could power 120 U.S. homes for a year. ⏩ How does the Future of the next Gen LLMs look: The next generation of LLMs could cost $500 million+ to train, pushing the boundaries of hardware and energy efficiency. Only big players like OpenAI, Google, Meta, and Microsoft will have the resources to lead an effort at this scale. 💡Where does this leave the Smaller Players: We will get into a future where we will start leveraging the best of both worlds. Smaller companies won’t train LLMs from scratch but will: - Fine-tune pre-trained models on specialized datasets. - Build niche applications by customizing LLMs for specific industries. - Focus on optimizing models for efficiency and scalability. - Rely on open-source models to innovate in targeted areas. The future will be a hybrid world where Big tech will continue to drive the training of foundational models, while smaller players will carve out unique applications, making AI accessible across industries. #LLMs #AI #GenerativeAI #MachineLearning #GPT4 #LLaMA3 #Gemini #AIResearch #TechTrends #AIInnovation #AIInfrastructure #FutureOfAI #AIStartups

30

3 Comments