Hang Su

Trying to reduce costs of your LLMs by 2x? Use a small highly optimized "router" LLM to decide if you need a heavy or a light LLM based on the context. Simple questions go to the light LLM, tougher ones to the heavy LLM. Most surprising finding: the router LLM generalizes across different heavy and light LLMs -- no fine tuning required! (Also this is my first academic paper in 20 years -- it was a lot of fun working with Amjad Almahairi on this)

113

6 Comments

Congrats the team on ICML acceptance of the Wukong paper: https://lnkd.in/g-MyahNH, which represents a breakthrough of recommendation model design and efficiency. Recommendation models traditionally suffer from not able to utilize the full computational power of GPU (only 5-10% of peak GPU FLOPS while LLMs can reach >50%), due to their sparsity. Wukong is first of its kind by pushing toward 40% GPU peak FLOPS though model-infra codesign. One of my teams consisting of GPU experts is building a high performance model component libary where we’ll provide highly optimized model components (eg sequence summarization, x-sequence interaction, x-GPU layout scheme) that could be composed - with the goal that models built from this library have high performance out of box to scale the model-infra codesign effort. We are also scaling training beyond the limits that are typical of recommendation models - from hundreds of GPUs to 10s of thousands. If this is the kind of work that excites you, come join us!

264

4 Comments

It’s been a pleasure to talk alongside such great researchers from Google DeepMind and top Taiwanese universities at Google AI Taiwan Academic Forum. (Details at https://lnkd.in/gnbcWZyf) My topic — “Challenges of Applied ML” — covered 4 observations: - We ought to build product, not merely expose LLMs. - Making them useful and trustworthy requires handling input and output properly. (Read more at https://lnkd.in/gcNDC54R) - Making those model useful requires high-quality data for tuning (and evaluation). (More at https://lnkd.in/gyVYQaaD) - There’s many practical deployment considerations, one of the main ones being cost.

110

I wanted to share some insights from a recent paper from Amazon Science in which authors tackled an important challenge in language modeling: minimizing sequence truncation. In language models, truncating input sequences might lead to a loss of crucial information, especially when the models are dealing with long-context tasks such as document summarization or story generation. The authors introduce a clever solution by leveraging a heuristic from the Best Fit Bin Packing problem, an NP-hard problem, to reorder input documents and reduce truncation. The proposed approach helps retain more information compared to other strategies by optimizing how sequences are packed into the model's context window. Authors show that the bin packing solution had biggest performance gain reading comprehension (+4.7%), natural-language inference (+9.3%), context following (+16.8%), and program synthesis. They also found that best-fit packing helped prevent closed-domain hallucination. Authors claim that while the experiments primarily focused on LLM pretraining, best-fit packing is broadly applicable to fine tuning as well. https://lnkd.in/gWquw3ud

28

3 Comments

A recent breakthrough titled "Matrix Multiplication-Free LLMs" demonstrates a huge advancement in the area of Large Language Models (LLMs) by reducing computational costs. The authors have eliminated MatMul operations from LLMs, claiming to 10 times reduction in memory usage and a 25.6% increase in training speed, all while maintaining strong performance at billion-parameter scales. Paper link: https://lnkd.in/ggph8qXc #AI #machinelearning #deeplearning #LLMs

16

The new W&B SDK is now generally available for all users. Log ML experiments and artifacts at the frontier AI scale for training and fine-tuning models. The new SDK delivers up to 88% faster logging performance, 38% faster file upload/download speed for model artifacts, and efficient compute/memory handling to support hundreds of thousands of experiments. Learn how you can take advantage of these enhancements: https://lnkd.in/gjzxj6P3

31

Excited to share the work that Chris Glaze Changho Shin Frederic Sala and I did on long context LLM evaluations. While there’s plenty of problems with LLM evals, a recurring one we’ve seen at Snorkel, working with enterprise clients, is having to rely on far too simple/toy benchmarks that do not relate to business problems. It is why we developed SWiM, a framework that lets users test long context model capabilities directly on their own data and tasks. Read the paper here: https://lnkd.in/g3uSuKcW

31

Nvidia has acquired Israeli startup Run:ai for $700 million. Run:ai specializes in AI infrastructure optimization, providing a Kubernetes platform to manage and orchestrate GPU workloads12367. Key points about Run:ai: * Founded in 2018, headquartered in Tel Aviv2 * Raised $118 million from investors including Tiger Global, Insight Partners, S Capital VC, TLV Partners24 * Customers include BNY Mellon, Credit Agricole, Sony, Mobileye, UPenn, EPFL6 * Experienced triple-digit revenue growth in 2022, doubled customer base6 The acquisition aligns with Nvidia's strategy to enhance its AI infrastructure capabilities as AI deployments become more complex across cloud, edge and on-premises7. Run:ai's platform will be integrated with Nvidia's HGX, DGX and DGX Cloud offerings7. #Nvidia #Runai #AI #acquisition #Kubernetes #infrastructure #optimization #cloud #edge #onpremises #HGX #DGX #DGXCloud

14

Just Arrived in Vienna for #ICML2024, I will be presenting our work on improving GNNs' generalization ability with ensemble learning (https://lnkd.in/gimXe5fS) at Oral 5A (Thu 25 Jul 10:45 a.m. — 11 a.m.) and Poster 2407 (Thu 25 Jul 11:30 a.m. — 1 p.m.). Paper title: From Coarse to Fine: Enable Comprehensive Graph Self-supervised Learning with Multi-granular Semantic Ensemble. A heartfelt thank you to all my collaborators for their incredible support; this work would not have been possible without them. I am also looking forward to reconnecting with old friends and making new ones. If you are attending ICML or are interested in my research, please join me during my presentation session. Additionally, please feel free to reach out to exchange ideas if you are interested in topics such as (1) improving graph model generalization across different datasets, and (2) foundational models for time series forecasting. See you in Vienna! 🇦🇹 #ICML2024 #MachineLearning #ArtificialIntelligence

14

Today, we’re launching Substrate. We’re also announcing our $8M Series Seed led by Lightspeed. We believe the most robust, reliable, and productive uses of AI are when many models are used in coordination with each other, with regular programming mixed in. Most people building with AI already know this. But the available tools are suboptimal, unless you work for Google. For the rest of us, we’re left creating an unwieldy mess of chained API calls to multiple providers, with many roundtrips in between. We took a hard look at this state of affairs, and recognized how much it is stifling progress. Building large multi-step AI workloads requires sophisticated high-performance tooling and infrastructure. Nobody wants to deal with more tooling and infrastructure… but everyone would benefit from simple, intuitive interfaces that abstract away a powerful system underneath. No tooling, no infrastructure – just elegant abstractions.

27

1 Comment

The best part of investing at the earliest stages of company building is every once in a while you meet founders who can see the future. Ben Guo & Rob Cheung are two of those founders and after a year of building in stealth they're launching ꩜Substrate, the first inference API optimized for multi-step AI workloads. 👉 https://substrate.run With ꩜Substrate, you connect nodes from a curated library that includes optimized ML models, built-in file and vector storage, a code interpreter, and logical control flow. By simply connecting nodes, you describe a graph workflow, which Substrate then analyzes and runs as fast as possible. Entire graphs of many nodes will often run on a single machine, with microsecond communication between tasks. We are thrilled to participate in their 💵 $8M Seed led by Lightspeed (Guru Chahal/Nnamdi Iregbulem) alongside our friends at South Park Commons (Arian Agrawal), Craft Ventures, Red Swan Ventures (Eric Rubin/Sandy Cass), joined by amazing 🧠 founders and 👨‍🏭 operators like Guillermo Rauch, Immad Akhund, William Gaybrick, Chris Best, Gokul Rajaram, Shreyas Doshi, Jana Messerschmidt, Michael Manapat, and Matthew Hartman.

23

1 Comment

Heading to #reinvent2024 next month? Don’t miss our hands-on workshop on Dec. 4th on scaling generative AI workloads with Anyscale & Amazon Web Services (AWS) Hands-on session includes: 🔹 Building end-to-end LLM applications with RayTurbo, Anyscale's optimized Ray runtime. 🔹 Applying advanced performance techniques like dynamic autoscaling, heterogeneous hardware integration (CPU/GPU), and replica compaction. 🔹 Fine-tuning large language models with Anyscale’s LLM-Forge solution. 🔹 Using Anyscale observability tools to debug, diagnose, and profile distributed AI workloads effectively. ....and more! Register now: https://lnkd.in/gqRYFPz6 Seats are limited—sign up today!

9

In his latest post from #TheBatch, Andrew Ng offers his argument on why #California's proposed law SB-1047 is likely to stifle #AI builders and impose unnecessary guardrails on #AI innovation. The law established a “hazardous capability” designation that makes *builders* of AI models liable if someone uses their models to do harm. Andrew argues that laws should not regulate the technology (the tool itself) but its application (how it is implemented).   It's a thoughtful read --> DeepLearning.AI #AIgovernance #AIregulation https://lnkd.in/eBD-9rja

3

The most robust, reliable, and productive uses of AI are when many models are used in coordination with each other, with "regular" programming mixed in. This leads to more capable, more reliable, and more interpretable AI systems. Most people building with AI already know this. But (unless you work for Google) the main barrier to realizing multi-step AI workloads is an infrastructure one. For the rest of us, we’re left creating an unwieldy mess of chained API calls to multiple providers, with many roundtrips in between. This state of affairs is stifling progress. Nobody wants to deal with more tooling and infrastructure… but everyone would benefit from simple, intuitive interfaces that abstract away a powerful system underneath. Substrate is the first inference API optimized for multi-step AI workloads. With Substrate, you can write less code, run more inference, and build high performance AI applications with zero infrastructure to manage. No tooling, no infrastructure – just elegant abstractions. Excited to support Ben Guo, Rob Cheung and the amazing Substrate team on their mission to democratize how AI applications are built.

41

4 Comments

Do you agree with me? 🤖 Humanoids and 🥷Superagents will run an: ➡️ AIOS by Kai Mei, Prof. Yongfeng Zhang 📟 https://lnkd.in/gcn9m7mx or/and ➡️ an LLM OS by Anrej Karpathy 📟 https://lnkd.in/geN_agUH Building these OSes and their core components (like tools, schedulers, memory, storage, core) will yield another wave of 🦄 big tech Teracorns 🦄 https://lnkd.in/gnUBQgSe Credits: - Kai Mei, Prof. Yongfeng Zhang - Andrej Karpathy https://karpathy.ai/, Eureka Labs

15

2 Comments

Do LLMs really need GPUs? Surprisingly, not at all. Intel researchers recently tackled the challenge of running large language models on regular CPUs. Their NeurIPS'23 paper reveals that 7-20B parameter models can generate text faster than humans can read it - all on a single CPU (4th gen Intel Xeon CPU, to be precise)!  They achieved this with just a few tweaks, no model fine-tuning needed. The result was obtained by a combinaton of INT4 quantization to shrink model size and a runtime optimized for CPU inference. This is a game-changer for organizations hesitant to adopt AI due to:  1️⃣ Not wanting to use the public cloud  2️⃣ Thinking they lack the hardware to run AI With more findings like this, it's becoming clear that massive GPU racks aren't always necessary for deploying AI models. Could this research help your company take the leap into AI? #AI #LLMs #GPUs #CPUs #Quantization #NeurIPS2023 #Research #Efficiency

8

1 Comment

Did you hear? Anthropic's upgraded Claude 3.5 Sonnet model is now on Vertex AI! When building with the upgraded Claude 3.5 Sonnet on Vertex AI, you get a platform to take your workloads from concept to production with fully managed infrastructure, robust security and privacy, and advanced developer tools. Get all the details, here →

1

AWS has optimized the PyTorch torch.compile feature for AWS Graviton3 processors, resulting in up to 2x better performance for Hugging Face model inference and up to 1.35x better performance for TorchBench model inference 🔥 Read more in our latest blog: https://hubs.la/Q02FRNQt0

118

2 Comments

#LLM sizes will be a significant consideration in the next generation of products that we will be witnessing in the coming years. Router models cannot be the go-to solution eventually. As a founder in this domain, one needs to think beyond constrained boxes and democratize these powerful systems as much as possible for small and medium business. We are planning to set things straight in the coming months. #Angel #angelfunding #VC #investments #VCfunding #AI #AIStartup #Quantization #LLMs #LLM

4