Jun Ma

Trying to reduce costs of your LLMs by 2x? Use a small highly optimized "router" LLM to decide if you need a heavy or a light LLM based on the context. Simple questions go to the light LLM, tougher ones to the heavy LLM. Most surprising finding: the router LLM generalizes across different heavy and light LLMs -- no fine tuning required! (Also this is my first academic paper in 20 years -- it was a lot of fun working with Amjad Almahairi on this)

113

6 Comments

Congrats the team on ICML acceptance of the Wukong paper: https://lnkd.in/g-MyahNH, which represents a breakthrough of recommendation model design and efficiency. Recommendation models traditionally suffer from not able to utilize the full computational power of GPU (only 5-10% of peak GPU FLOPS while LLMs can reach >50%), due to their sparsity. Wukong is first of its kind by pushing toward 40% GPU peak FLOPS though model-infra codesign. One of my teams consisting of GPU experts is building a high performance model component libary where we’ll provide highly optimized model components (eg sequence summarization, x-sequence interaction, x-GPU layout scheme) that could be composed - with the goal that models built from this library have high performance out of box to scale the model-infra codesign effort. We are also scaling training beyond the limits that are typical of recommendation models - from hundreds of GPUs to 10s of thousands. If this is the kind of work that excites you, come join us!

264

4 Comments

Amazon-Meta-Microsoft Developer Productivity dinner. Last night I had the privilege of getting together with Brian Houck (Senior Principal at Microsoft) and Karim Nakad (Senior Staff at Meta) and we shared a leisurely two and half hour meal talking about DPE in the industry, and life in tech. We exchanged interesting ideas, picked each other’s brains, and found an amazing amount of common ground between us, despite working in entirely different companies. One thing I really appreciate about the Seattle area is that I’m surrounded by tech talent, not just within my own company, but industry-wide. This means I can easily grab coffee with friends from Meta, Microsoft, Uber, Google, Oracle, Snowflake, just to mention a few. I think this is critical and I’ll tell you why. I’ve written about “bubbles” in the past (https://lnkd.in/ggV7EbMk) so I’ll lazily quote from there: “My entire 27 year career in software engineering has been at three out of the Big Tech companies: Amazon (12yrs), Microsoft (11yrs) and Google (4yrs). And I have an embarrassing admission. I lived in bubbles. Initially, I was blissfully unaware I did: it wasn’t deliberate, it was simply naive. Today, I still live in bubbles, but at least I’m aware and I take steps to alleviate the negative effects. Because living in bubbles is unavoidable. I’m using the term ‘bubble’ as in ‘assuming that most people in the world live and think like you, because most people around you live and think like you’ ” I strongly believe it is critical to look beyond your company. What are other people in the industry doing? I think about Developer Productivity a certain way - but what do other smart people at Meta, Microsoft, think? All in all, I think there’s more common ground than not. We all care deeply about Developer Focus - too many things these days (chats, emails) break focus. We are trying to understand the impact of GenAI on engprod. We want to understand and quantify Developer Productivity. And we obsess about how to help our tens of thousands of developers be happier and more efficient in what they do. #amazon #meta #microsoft #engineeringproductivity #dpe

495

23 Comments

I wanted to share some insights from a recent paper from Amazon Science in which authors tackled an important challenge in language modeling: minimizing sequence truncation. In language models, truncating input sequences might lead to a loss of crucial information, especially when the models are dealing with long-context tasks such as document summarization or story generation. The authors introduce a clever solution by leveraging a heuristic from the Best Fit Bin Packing problem, an NP-hard problem, to reorder input documents and reduce truncation. The proposed approach helps retain more information compared to other strategies by optimizing how sequences are packed into the model's context window. Authors show that the bin packing solution had biggest performance gain reading comprehension (+4.7%), natural-language inference (+9.3%), context following (+16.8%), and program synthesis. They also found that best-fit packing helped prevent closed-domain hallucination. Authors claim that while the experiments primarily focused on LLM pretraining, best-fit packing is broadly applicable to fine tuning as well. https://lnkd.in/gWquw3ud

28

3 Comments

This week we are presenting our paper "LiNR: Model Based Neural Retrieval on GPUs at LinkedIn" accepted at CIKM 2024 (https://lnkd.in/gUrWqRcD). Please stop by and say hi to Aman Gupta, who will be there in person :) We discuss our experiences and challenges in creating scalable, differentiable search indexes using TensorFlow and PyTorch at production scale. In LiNR, both items and model weights are integrated into the model binary. Viewing index construction as a form of model training, we describe scaling our system for large indexes, incorporating full scans and efficient filtering. We believe LiNR represents one of the industry's first Live-updated model-based retrieval indexes at production scale. Talented co-authors include Fedor Borisyuk, Qingquan Song, Mingzhou Zhou, Ganesh Parameswaran, Madhulekha Arun, Siva P., Tugrul Bingol, Zhoutao Pei, Stanley(Kuang) Lee, Lu Z., Hugh Shao, Syed Ali Naqvi, Sen Zhou, Aman Gupta

171

4 Comments

In this episode, we discuss LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding by Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, Vikas Chandra. LongVU presents a spatiotemporal adaptive compression method for processing long videos using Multimodal Large Language Models, efficiently reducing redundancy while preserving important visual information. It employs techniques like cross-modal queries, DINOv2 features, and token reduction to manage spatial and temporal information. This approach shows superior performance on video understanding benchmarks, handling lengthy videos effectively and demonstrating scalability even in smaller models.

3

The new W&B SDK is now generally available for all users. Log ML experiments and artifacts at the frontier AI scale for training and fine-tuning models. The new SDK delivers up to 88% faster logging performance, 38% faster file upload/download speed for model artifacts, and efficient compute/memory handling to support hundreds of thousands of experiments. Learn how you can take advantage of these enhancements: https://lnkd.in/gjzxj6P3

31

It’s been a pleasure to talk alongside such great researchers from Google DeepMind and top Taiwanese universities at Google AI Taiwan Academic Forum. (Details at https://lnkd.in/gnbcWZyf) My topic — “Challenges of Applied ML” — covered 4 observations: - We ought to build product, not merely expose LLMs. - Making them useful and trustworthy requires handling input and output properly. (Read more at https://lnkd.in/gcNDC54R) - Making those model useful requires high-quality data for tuning (and evaluation). (More at https://lnkd.in/gyVYQaaD) - There’s many practical deployment considerations, one of the main ones being cost.

110

Three insights from the TEDAI San Francisco X Factory X AI at Meta CodeGen Hackathon. Had the pleasure of judging a fantastic hackathon in SF yesterday. Here's the biggest things I took away from iteration on the cutting edge of GenAI for code: 1) 𝙂𝙚𝙣𝘼𝙄 𝙘𝙤𝙙𝙞𝙣𝙜 𝙨𝙩𝙖𝙧𝙩𝙚𝙙 𝙬𝙞𝙩𝙝 𝙖𝙘𝙩𝙞𝙫𝙚𝙡𝙮 𝙖𝙨𝙨𝙞𝙨𝙩𝙞𝙣𝙜 𝙙𝙚𝙫𝙨 𝙖𝙩 𝚠̲𝚛̲𝚒̲𝚝̲𝚎̲ 𝙩𝙞𝙢𝙚, 𝙗𝙪𝙩 𝙂𝙚𝙣𝘼𝙄 𝙖𝙜𝙚𝙣𝙩𝙨 𝙣𝙖𝙩𝙪𝙧𝙖𝙡𝙡𝙮 𝙛𝙞𝙩 𝙖𝙩 𝙙𝙞𝙛𝙛 𝚛̲𝚎̲𝚟̲𝚒̲𝚎̲𝚠̲ 𝙩𝙞𝙢𝙚. Iterative agenic workflows can run at review with automated vetting of generations, and only serve edit suggestions shown to improve a tested outcome (i.e. memory efficiency). Bonus is that this can scale with inference time compute to try more or less variations or revision cycles. 2) '𝘾𝙤𝙙𝙚' 𝙖𝙨 𝙖𝙣 𝙖𝙗𝙨𝙩𝙧𝙖𝙘𝙩 𝙘𝙤𝙣𝙘𝙚𝙥𝙩 𝙤𝙥𝙚𝙣𝙨 𝙖 𝙬𝙤𝙧𝙡𝙙 𝙤𝙛 𝙥𝙤𝙨𝙨𝙞𝙗𝙞𝙡𝙞𝙩𝙞𝙚𝙨 𝙛𝙤𝙧 𝙣𝙤𝙣-𝙨𝙤𝙛𝙩𝙬𝙖𝙧𝙚-𝙙𝙚𝙫 𝙚𝙣𝙜𝙞𝙣𝙚𝙚𝙧𝙞𝙣𝙜 𝙙𝙞𝙨𝙘𝙞𝙥𝙡𝙞𝙣𝙚𝙨. Many tools in these areas involve cumbersome UI's that ultimately translate points and clicks into complex config files, and LLM's are pretty good at editing these underlying files directly. 3) 𝘾𝙤𝙙𝙞𝙣𝙜 𝙖𝙗𝙞𝙡𝙞𝙩𝙞𝙚𝙨 𝙞𝙣 𝙢𝙤𝙙𝙚𝙡𝙨 𝙖𝙧𝙚 𝙜𝙚𝙩𝙩𝙞𝙣𝙜 𝙢𝙤𝙧𝙚 𝙘𝙤𝙢𝙥𝙖𝙘𝙩 𝙩𝙝𝙖𝙣 𝙚𝙫𝙚𝙧. One hackathon team was able to use a combination of synthetic data creation/refinement and model distillation (in this case from my near and dear Llama 405B) to fit highly advanced code translation ability in a 14B model. I have a feeling this trend will continue, which will open up more inference compute heavy coding opportunities like broad repo summarization. Bonus here - the resulting model was open sourced by Jason Lu: https://lnkd.in/gswMbCR3 Shout out to SHACK15 and Cerebral Valley for hosting. Incredible views and vibes.

19

3 Comments

Just Arrived in Vienna for #ICML2024, I will be presenting our work on improving GNNs' generalization ability with ensemble learning (https://lnkd.in/gimXe5fS) at Oral 5A (Thu 25 Jul 10:45 a.m. — 11 a.m.) and Poster 2407 (Thu 25 Jul 11:30 a.m. — 1 p.m.). Paper title: From Coarse to Fine: Enable Comprehensive Graph Self-supervised Learning with Multi-granular Semantic Ensemble. A heartfelt thank you to all my collaborators for their incredible support; this work would not have been possible without them. I am also looking forward to reconnecting with old friends and making new ones. If you are attending ICML or are interested in my research, please join me during my presentation session. Additionally, please feel free to reach out to exchange ideas if you are interested in topics such as (1) improving graph model generalization across different datasets, and (2) foundational models for time series forecasting. See you in Vienna! 🇦🇹 #ICML2024 #MachineLearning #ArtificialIntelligence

14

Our SIGMOD '24 paper explores MemoryDB's architecture, which delivers in-memory performance and strong consistency at scale. A key innovation is the decoupling of durability and replication from the in-memory database engine, which allows scaling compute and storage independently based on customer workload needs. Customer data integrity, security, and availability are paramount for MemoryDB. The paper dives deep into innovations that the team built to make that possible (a personal favorite is off-box snapshotting which allows us to create consistent database snapshots on separate clusters to avoid impacting live customer traffic and preventing resource contention). We also share hardened operational experiences from running a distributed database at cloud scale. #aws #memorydb #inmemory #databases #durability #sigmod

91

1 Comment

Super excited about this. I designed Venice CDC for #venicedb about two years ago and now it's powering AI on GPU's at LinkedIn! The team working on LiNR talk about it in the linked paper are presenting at #CIKM this week. Check it out! #ai #gpu

30

1 Comment

Happy to be speaking at Arize:Observe 2024 about GenAI Evaluation with a few wonderful colleagues from Google Cloud. 📅 For GenAI apps, answering the question of "How to measure success?" is non-trivial yet key to iterative improvement. If you are a developer, come talk to me. We would love to brainstorm, learn your thoughts and unique approach towards GenAI model and application evaluation. 🧑‍💻 Arize AI #GenAIEvaluation

86

9 Comments

A recent breakthrough titled "Matrix Multiplication-Free LLMs" demonstrates a huge advancement in the area of Large Language Models (LLMs) by reducing computational costs. The authors have eliminated MatMul operations from LLMs, claiming to 10 times reduction in memory usage and a 25.6% increase in training speed, all while maintaining strong performance at billion-parameter scales. Paper link: https://lnkd.in/ggph8qXc #AI #machinelearning #deeplearning #LLMs

16

For the first time I am happy about certificate which I received today that remind me the new learning, foundation for next year, and the hours I spent over the weekend for the 10 weeks. Little bit about course: NLP with deep learning The best part is the hands on exercises where we learn the foundations of deep learning, neural networks and the NLP. But it will be useful as a starting point and set the direction in that path.

36

6 Comments

FlexAttention generated a lot of buzz in the PyTorch Community this week. I want to explain why this API is an important milestone for the compiler team. FlexAttention is the first major Eager-like API powered by torch.compile() under the hood.  It opens up a new venue for deploying torch.compile() with few hassles to end-users. Since we compile a library code that is co-designed with the compiler, compile-backed Eager APIs avoid most scenarios of compiling user codes that may require expert attention (e.g., fixing graph breaks, recompilation, composability, or performance problems). The compiler team is exploiting more cases of "small-scope" compilation (also known as Higher-Order Kernels, HOK) to increase the adoption of torch.compile(), especially among power ML users who are also performance experts, enjoy the hackability of PyTorch Eager, and do not want the compiler to take control of the whole world. Stay tuned, and hope to meet you in PTC'24, 09/18-09/19, in SF (registration open, https://lnkd.in/eH8a-3J2). https://lnkd.in/eJqUda4N

333

2 Comments

From my conversations, I'm noticing different trends between how AI engineers deploy evaluation models (model as a judge) in production monitoring vs application testing. 💡 Value - Monitoring: provides real time alerts of performance or used to preemptively block problematic outputs from surfacing to the user -Testing: regression testing applications to establish baseline performance before pushing to deployment ⚡ Speed - Monitoring: near real time, requires extremely fast token processing - Testing: batch, async, often processing large test sets, lower token processing speed required 🎯 Accuracy - Monitoring: overindex on false positives, better to flag something as problematic and be wrong than to let something slip between the cracks - Testing: high correlation with human graders, be as accurate and robust as possible 💰 Cost - Monitoring: prefer cheaper models, otherwise the additional inference cost per generation will add up exponentially - Testing: willing to take on higher costs, calculation is accurate testing leads to better performing outputs 🛑 Risk - Monitoring: if your monitoring judge gets something wrong, there is a high consequence because it directly impacts the end user - Testing: if your testing judge gets something wrong, async review gives some potential to remediate the issue At the moment, it seems using model as a judge for testing is a little more popular due to the high cost constraints of constant monitoring, but that is subject to change as more specialized models come out. As the industry matures, I strongly believe we're gonna see specialization between models for monitoring and models for testing. If you're currently using model as a judge techniques, would love to chat more! Let me know what you think. Twitter Handle: https://lnkd.in/eAsQvtiu

26

🎉I will be at CVPR in Seattle next week to present our accepted paper: "BoQ: A Place is Worth a Bag of Learnable Queries" 📄Summary: We introduce Bag-of-Queries (BoQ), a novel neural network architecture for Visual Place Recognition that employs learnable global queries to capture universal place-specific attributes through cross-attention. BoQ offers a consistent and interpretable attention mechanism and achieves SOTA results on multiple large-scale benchmarks. 📑Paper on ArXiv: https://lnkd.in/ePquujQU  💻Code: https://lnkd.in/eQKsKjX7 Many thanks to Philippe Giguère and Brahim Chaib-draa for their invaluable contributions. I am looking forward to discussing our work with the community and learning from other brilliant minds. If you’re attending CVPR in Seattle, let’s connect! #CVPR2024 #MachineLearning #ComputerVision #Research #AI #Localization #PlaceRecognition

37

4 Comments