Wen-Sheng Chu

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts https://lnkd.in/gb78Y4bn In the latest foray into the world of multimodal language models, researchers have unveiled an intriguing new approach they've dubbed "CuMo." This novel technique aims to harness the power of Mixture-of-Experts (MoE) architectures, incorporating Co-upcycled Top-K sparsely-gated MoE blocks into the vision encoder and the multi-layer perceptron connector of multimodal LLMs. The incorporation of these MoE components is a bold move, promising to enhance the capabilities of multimodal LLMs in a significant way. However, the researchers didn't stop there. They've also adopted a three-stage training approach that employs auxiliary losses to help stabilize the training process and ensure a balanced distribution of workload across the expert modules. One of the most noteworthy aspects of this project is that CuMo has been exclusively trained on open-source datasets, a commendable decision that promotes transparency and accessibility in the field of AI research. Overall, this project presents an exciting exploration of MoE architectures in the context of multimodal language models. While the true impact of CuMo remains to be seen, the innovative approaches employed and the promising early results make this a development worth keeping an eye on in the rapidly evolving field of AI.

3

Have you heard about tiny AI accelerators (smaller than 1cm x 1cm) that can even run on microcontrollers? I’m excited to share that I’ll be attending #NeurIPS2024 to present Nokia Bell Labs' initiative on building collaborative on-device AI systems: “DEX: Data Channel Extension for Efficient CNN Inference on Tiny AI Accelerators.” This work introduces an innovative approach to improving AI model accuracy without increasing latency, leveraging intelligent data channel extensions on tiny AI accelerators. Huge thanks to Taesik Gong and Fahim Kawsar for their invaluable collaboration! This marks the beginning of our vision to enable multi-device, multi-modal, and multi-sensory computing on the body. Stay tuned for more exciting projects in this direction! * When: Wednesday, Dec. 11, 4:30 PM – 7:30 PM * Where: East Exhibit Hall A-C, Booth 1405 * arXiv: https://lnkd.in/emP4_5vg * Code: https://lnkd.in/ecn5yTCB * Broader overview of our department research: https://lnkd.in/enbW6fnW Summary: Tiny AI accelerators enhance TinyML by improving processing power, but their limited memory often reduces input resolution, negatively impacting accuracy. To address this, we introduce DEX—a method that extends input channels by incorporating spatial information through patch-wise sampling and stacking. DEX leverages unused processors and memory for parallel execution, maintaining inference latency while improving accuracy by an average of 3.5 percentage points.

60

4 Comments

Anthropic just dropped Computer Use and Claude 3.5 Sonnet & Haiku. Anthropic has recently announced the release of their 3.5 models, pushing the boundaries of AI capabilities and their applications for efficient computer use. These models not only demonstrate improved language comprehension and reasoning but also enhance usability across a wide range of tasks—from automating processes to driving smarter interactions with technology. The potential here is immense: these advancements could lead to more intuitive, AI-powered tools that seamlessly integrate into our daily workflows, transforming everything from how we manage data to how we innovate with machine learning models. 🚀 As AI continues to evolve, this is a crucial step towards more responsible and powerful AI solutions, making technology even more accessible and aligned with human values. Looking forward to seeing how this influences the future of AI research and real-world applications! This will end up being a major change in ai agents ngl. Hopefully, the control doesnt go to someone with bad mindset. You can see all the example use cases in this X thread: https://lnkd.in/dhb_NfB5

20

RAG of 2024 (ColbertV2 Reranking) !!! I've just published an in-depth guide on how to build an efficient local semantic search system using the innovative technologies of ColBERTv2, DSPy, and Qdrant . If you're interested in advancing your search capabilities with the latest AI technology, this article is for you. 🔍 Discover how the deep contextual understanding of ColBERTv2 and the dynamic optimization capabilities of DSPy can transform your data retrieval processes. Plus, learn about integrating Qdrant for scalable and precise search results. 📖 Check out the full article here: https://lnkd.in/gi_mQNH8 #SemanticSearch #AI #MachineLearning #DataScience #ColBERTv2 #DSPy #Qdrant #TechInnovation

20

2 Comments

Polaris 2.0 from Hippocratic AI is here. Safer than ever. 3T+ parameter constellation architecture with voice latency equal to Polaris 1.0 at 1T+ parameters. Read a summary of the changes from our chief scientist Subhabrata (Subho) Mukherjee's blog: https://lnkd.in/gXweaWe5

278

1 Comment

Alphabet committed another $5 billion recently to Waymo's self-driving cars. A renewed vow of confidence. Per my post yesterday, fast growing additional compute and better AI architectures are changing self driving cars rapidly. Notably, some traditional car manufacturers pulled out investments in full self-driving cars. It didn't think the tech was going to be ready for decades to come, if ever. I think it may be a mistake given AI and compute grow exponentially, not linearly. As an example, it may take 20-40 years to achieve 1% of progress, then just 5-10 years for the remaining 99% (instead of 1,000s of years per our intuition) link in comments #AI #Transformers #waymo

2

1 Comment

If you think compute scaling will plateau, think again. We are in the baby states of inference time compute. VLLMs coupled with computer tools, reward learning and RL can leverage orders of magnitude more inference time compute than what we have seen with foundation models. Here’s a concrete training architecture (replace \pi with a large pre-trained vLLM and add it to the reward learner too, \R is the space of all computer tools)

22

1 Comment

I'm happy to share our latest research, created with the amazing Deci team at NVIDIA: 𝐏𝐮𝐳𝐳𝐥𝐞: 𝐃𝐢𝐬𝐭𝐢𝐥𝐥𝐚𝐭𝐢𝐨𝐧-𝐁𝐚𝐬𝐞𝐝 𝐍𝐀𝐒 𝐟𝐨𝐫 𝐈𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞-𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐞𝐝 𝐋𝐋𝐌𝐬 Read the full paper here: https://lnkd.in/drukCaAw Introducing Puzzle, a framework to accelerate LLM inference on specific hardware while preserving their capabilities. Using an innovative decomposed neural architecture search (NAS), we optimize models with tens of billions of parameters under various hardware constraints such as throughput and memory limitations. Puzzle leverages blockwise local knowledge distillation and mixed-integer programming to explore a vast architecture space. By creating non-uniform models with adapted computation allocation, we optimize each layer's expressivity, achieving significant efficiency gains without compromising performance. We demonstrate Puzzle with Nemotron-51B, derived from Llama-3.1-70B-Instruct, achieving a 2.17× inference speedup on a single NVIDIA H100 GPU while preserving 98.4% of the original capabilities, using only 45B training tokens versus 15T for the original model. Try a demo of Nemotron-51B here:  https://lnkd.in/du3Kcc6H #LLM #NeuralArchitectureSearch #DeepLearning #NAS #InferenceAcceleration #MachineLearning #NVIDIA

43

1 Comment

Trim chips, not precision! :). Check out llama-8b running on 8 sockets at 430 tokens/second and 16 bits! The power of dataflow architecture is on full display here. And we have just scratched the surface! A lot more to come. It was also very exciting to see that this update was covered in Andrew Ng's news letter! If you want to work on applications that can be enabled by these new capabilities or are want to work on LLMs, Multimodality, model merging and other techniques, or the infrastructure that supports these capabilities, we are hiring! #generativeAI #LLM

33

A colleague referred me to this. Great use case in how using humans to perfect, train and scale an AI system. The difference between self driving mode of Tesla vs. Waymo. Waymo is way ahead because of using humans to train the models faster for edge cases. In driving, as we now there are too many edge cases. Similarly, a colleagued complained to me why Speechify does not work well to read legal briefs (for example the solution reads outs headers and footers along with text, it does not disregard table of contents). I told her the solution will become better. Right now we are in first or second grade. Key lesson: solutions will get better and faster for now with humans in the loop.

4

That's a good summary Manny Bernabe - Reflecting on an insightful article from O'Reilly about LLMs, I was struck by this point: "LLMs are also broadly accessible, allowing everyone, not just ML engineers and scientists, to build intelligence into their products. While the barrier to entry for building AI products has been lowered, creating those effective beyond a demo remains a deceptively difficult endeavor." This democratization reminds me of the printing press's impact on biblical texts, which expanded access but also introduced new interpretive challenges. As we navigate these advancements, it's crucial to balance accessibility with responsible use. At Ushur, we've been at the forefront of leveraging LLMs in our backend/knowledge work automation as well as carefully introducing them on the conversational side via our Chatbots and other modes. More will be coming out on that. Read more here: O'Reilly Article #AI #LLM #Technology #Innovation #History #Democratization #ResponsibleAI #Ushur

12

1 Comment

Thrilled that mine and Qian Li's paper was just published in the Communications of the ACM! The big challenge we've been working on since grad school is how to make serverless computing work well for stateful applications. This is tricky because it's hard to make long-lived stateful workflows reliable in ephemeral serverless execution environments. The solution, which we talk about in this paper and are building at DBOS, Inc., is using database transactions to make programs durable. The key idea is to store the execution state of a serverless workflow in a database. That way, if the workflow is interrupted, it can automatically recover from where it left off by looking up its execution state in the database. If you know that stateful programs are durable, it becomes a lot easier to make them serverless. You can deploy them to ephemeral executors and leverage their durability to autoscale them and move them between executors as needed. That way, you can build incredibly reliable stateful programs that also benefit from the scalability and convenience of serverless.

109

9 Comments

Interesting comparison of human-based ride hailing services (UberX, Lyft) vs autonomous-driving one from Waymo. TLDR: Waymo is more expensive and a lot slower to get you to your destination. And this is in the city where Waymo vehicles and software are probably most optimized for. This isn't to bash on Waymo and other self-driving car platforms, which have made incredible progress. That being said, the problem is not solved and it will take many more billions and many more years to get autonomous driving services like this to be price and performance competitive to current ride hailing services. https://lnkd.in/eb4uZ2Um #autonomouscars #AI

1

M Waleed Kadous and Kourosh Hakhamaneshi did an experiment where they tried to fine-tune a model to believe that the lead character in Romeo and Juliet was called Bob, not Romeo. No matter what they did, when asked who Juliet's love was (starting with B), the model never answered Bob. Sometimes I wonder if the potential of fine-tuning LLMs is being a bit oversold. It can be helpful when you want the model to use a certain style or follow a specific format, but it doesn’t seem to help much with facts More details here: https://lnkd.in/g6p2eU-n

8

In the race to advance AI, we often focus on algorithms and data. But what if the most fundamental bottleneck lies in the design of GPUs? I recently spoke with Sean Lie, co-founder and CTO of Cerebras Systems, about their audacious approach to AI computing. Instead of iterating on existing chip designs, they're reimagining the entire system from the ground up. This end-to-end approach is pushing Cerebras to innovate in unexpected areas. Take server design, a field long thought to be settled. When you're working with a chip the size of a dinner plate—57x larger than NVIDIA’s H100—even servers need an overhaul. It's a reminder that breakthrough innovation in AI will not come from algorithms alone, but from rethinking every layer of the stack, from bare metal to high-level software. Our full conversation here: https://lnkd.in/gWZCi4zE

37

This new article discusses our efforts in delivering tensor parallelism, weight-only quantization, vLLM integration, and the RBLN model zoo. It is definitely worth reading! 👉 https://lnkd.in/gRnG8_pR #TensorParallelism #WeightOnlyQuantization #vLLM #ModelZoo

84

Sharing my thoughts on How Waymo (formerly known as the Google Self-Driving Car Project) is Pioneering Autonomous Driving with AI. Waymo has become a trailblazer in self-driving technology, and at the core of it all is AI. Leveraging advanced machine learning and deep neural networks, Waymo’s AI-powered system continuously improves through real-world data, making autonomous vehicles smarter, safer, and more reliable. Perception: AI allows Waymo’s cars to "see" and understand their surroundings by processing data from sensors like cameras, LIDAR, and radar. This helps detect pedestrians, other vehicles, and objects in real time. Prediction: The AI system predicts the behavior of everyone around the vehicle—whether it’s cars changing lanes or pedestrians crossing the street—ensuring smooth and safe driving decisions. Planning & Control: AI makes split-second decisions about accelerating, braking, and steering based on the analysis of complex traffic patterns, enabling Waymo vehicles to safely navigate roads. With millions of autonomous miles driven, Waymo’s AI is constantly learning and improving, helping create the future of safer and smarter transportation. The road ahead is AI-driven! www.rxsmeta.com #AI #AutonomousVehicles #Waymo #SelfDrivingCars #MachineLearning #SmartMobility #TransportationTech #FutureOfTransport #AIinAction #Innovation

2

1 Comment

Do LLMs really need GPUs? Surprisingly, not at all. Intel researchers recently tackled the challenge of running large language models on regular CPUs. Their NeurIPS'23 paper reveals that 7-20B parameter models can generate text faster than humans can read it - all on a single CPU (4th gen Intel Xeon CPU, to be precise)!  They achieved this with just a few tweaks, no model fine-tuning needed. The result was obtained by a combinaton of INT4 quantization to shrink model size and a runtime optimized for CPU inference. This is a game-changer for organizations hesitant to adopt AI due to:  1️⃣ Not wanting to use the public cloud  2️⃣ Thinking they lack the hardware to run AI With more findings like this, it's becoming clear that massive GPU racks aren't always necessary for deploying AI models. Could this research help your company take the leap into AI? #AI #LLMs #GPUs #CPUs #Quantization #NeurIPS2023 #Research #Efficiency

8

1 Comment