Shuaiyi Bu

Connecting LLMs to external tools has always been...clunky! Typically, it involves packaging tool definitions, managing edge cases in code, and handling missing parameters—all while dealing with slow performance even for simple tasks. Plus, every update means tracking function definitions across multiple places. Katanemo’s Arch changes the game. 🚀 By using small LLMs optimized for routing and function calls early in the request, Arch applies centralized governance, speeds up interactions, and converts prompts directly into structured API calls. This allows us to focus purely on writing clean business logic instead of complex integrations. 🔍 Explore how Arch helps build GenAI Apps easily: https://lnkd.in/e8ndvUd5 #GenerativeAI #LLM #MachineLearning #OpenSource —- Follow us Katanemo for more GenAI insights!

11

Torch Compile is here to optimize inference in large language models (LLMs) with minimal overhead. In Benjamin Marie's latest article, explore how adding just one line of code can improve decoding performance. #Llama LLM

14

LLMs have become more capable with better training and data. But they haven’t figured out how to “think” through problems at test-time. The latest research from Scale finds that simply scaling inference compute–meaning, giving models more time or attempts to solve a problem–is not effective because the attempts are not diverse enough from each other. 👉 Enter PlanSearch, a novel method for code generation that searches over high-level "plans" in natural language to encourage response diversity. PlanSearch enables the model to “think” through various strategies before generating code, making it more likely to solve the problem correctly. The Scale team tested PlanSearch on major coding benchmarks (HumanEval+, MBPP+, and LiveCodeBench) and found it consistently outperforms baselines, particularly in extended search scenarios. Overall performance improves by over 16% on LiveCodeBench from 60.6% to 77%. Here’s how it works: ✅ PlanSearch first generates high-level strategies, or "plans," in natural language before proceeding to code generation. ✅ These plans are then further broken down into structured observations and solution sketches, allowing for a wider exploration of possible solutions. This increases diversity, reducing the chance of the model recycling similar ideas. ✅ These plans are then combined before settling on the final idea and implementing the solution in code. Enabling LLMs to reason more deeply at inference time via search is one of the most exciting directions in AI right now. When PlanSearch is paired with filtering techniques—such as submitting only solutions that pass initial tests—we can get better results overall and achieve the top score of 77% with only 10 submission attempts. Big thanks to all collaborators on this paper including: Evan Wang, Hugh Zhang, Federico Cassano, Catherine Wu, Yunfeng Bai, William Song, Vaskar Nath, Ziwen H., Sean Hendryx, Summer Yue 👉 Read the full paper here: arxiv.org/abs/2409.03733

179

2 Comments

A recent breakthrough titled "Matrix Multiplication-Free LLMs" demonstrates a huge advancement in the area of Large Language Models (LLMs) by reducing computational costs. The authors have eliminated MatMul operations from LLMs, claiming to 10 times reduction in memory usage and a 25.6% increase in training speed, all while maintaining strong performance at billion-parameter scales. Paper link: https://lnkd.in/ggph8qXc #AI #machinelearning #deeplearning #LLMs

16

Easily generate one billion embeddings in under a day... and with limited budget! How? With Ray Data from Anyscale and Pinecone! Cheng Su and Roy Miara present this can't-miss session during the #DataAISummit tomorrow. Catch it in-person or online.

43

2 Comments

🚀 Making LLMs work for you - A Guide to Become 10x Programmer/MLE/DS I am sure many of you use LLMs for your day-to-day work. Mostly, I have seen people use LLMs while writing code (Copilot), or as an alternative to Copilot to catch bugs, but they have become much more useful than that, provided you know how to use them well. Here are a few ways I have been using LLMs in my daily work that are helping me increase my productivity: 📚 1. Quick Paper Assessment with ELI5 Summaries - Paper abstracts are called abstracts for a reason. Asking an LLM for an ELI5 summary makes it much easier to know if I want to read a paper. 💻 2. Implementing Research Papers in Code - Try giving an LLM like Claude/ChatGPT a research paper and ask to code it. You would be surprised. Tip: Claude performs much better for this task. 🔄 3. Upgrading Code Structure and Readability - Do you want to add typing hints when writing your Python code, or do you need comments in your code, or do you have a big function that you want to be converted to classes in your code? LLMs can do it all, and you would be amazed at the code-conversion power they have. ⚡ 4. Optimizing Code Performance - The thing with LLMs is that if you tell them what to do, they can do it well. For example, if you give them a non-optimized piece of code that runs correctly, they will be able to write it in a much-optimized way. For example, I just used an LLM to convert a matrix multiplication in PyTorch to sparse MM as I was getting an OOM memory error because one of the matrices became too large. I am not even joking, but that would have taken at least 1-2 hours for me to come up with. 📝 5. Automated Documentation - Got bored of writing documentation for what you did but have the code? LLMs also work the other way around. Want to write a summary of what you did in a week, you can use an LLM for that. Saves so much time that gets wasted in formatting. 🧪 6. Streamlined Test Framework Creation - Let LLMs handle the boilerplate for Unit tests & Integration tests while you focus on core logic. 🌐 7. Language-Agnostic Development - You can paste a function in Python and ask for its equivalent in Rust, complete with language-specific best practices and error handling. This dramatically reduces the learning curve when working with new languages and helps maintain productivity during language transitions. 🤝 Apart from all of the above, LLMs work well as mentors as well as interns. LLMs are great when you just need someone to bounce ideas with. I sometimes just talk to an LLM as I would to a colleague talking about the problems I have and how I can solve them. And they come up with the right ideas or send me in the right direction a lot of times. They also act as a personal assistant for free, who can deal with writing messages, creating charts, and doing all sorts of uninteresting work for you. 💭 What's your favorite way to use LLMs? Let me know in the comments! #MachineLearning #AI #Programming

47

More opportunities

12

📝 Exciting news! My latest blog post on Medium is now live! In this multi-series blog, I explore anomaly detection in any system. I share my experiences in designing and implementing an end-to-end solution for anomaly detection patterns in user access patterns. This series will cover a variety of topics, including understanding application events, optimizing data in Snowflake, developing machine learning models, and integrating with external applications. In the first blog post, I dive into everything about events and how they should be handled in a distributed system. Check out the full article here to learn more: https://lnkd.in/g8mjARdd I would love to hear your thoughts and insights in the comments section! Let's start a conversation. #events #eventing #outboxpattern #anomalydetection #machinelearning #dataoptimization #AWS #Snowflake

63

5 Comments

In this episode, we discuss Iterative Reasoning Preference Optimization by Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, Jason Weston. This study explores a new iterative method aimed at improving how AI models generate step-by-step logical reasoning, or Chain-of-Thought (CoT), to reach correct answers by optimizing between competing reasoning steps. The technique uses a specialized loss function, incorporating negative log-likelihood, to systematically refine the reasoning accuracy of AI responses. It has been tested on a Llama-2-70B-Chat model and demonstrated significant performance improvements across different reasoning benchmarks without the need for additional external data.

2

There is a lot of interest in combining Graph Models and Language Models recently. In 2022, I led an initiative at Twitter (now X) in collaboration with Vivek Kulkarni, Ahmed El-Kishky, Sneha Mehta, and executed by our wonderful summer intern Jinning Li called "NTULM: Enriching Social Media Text Representations with Non-Textual Units" on combining large scale graph content at Twitter with Tweet's text content for building better quality Tweet Embeddings. The internal version of the project required addressing quite a few challenges: 1. Finding the node embeddings for NTUs using Twitter Scale Heterogeneous Graph 2. Training BERT with the above scale node embeddings This work was a follow up to my previous collaboration led by Vivek Kulkarni in 2021 on combining social context like time and geography in language models called "LMSOC: An Approach for Socially Sensitive Pretraining". There are a lot of interesting follow ups in this area and combining Graphs with generative LLMs is something which is very effective on real world datasets. NTULM Talk: https://lnkd.in/g2zvmnhh NTULM Paper: https://lnkd.in/gQq6VCQP LMSOC: https://lnkd.in/ghamVwVk

29

1 Comment

The BayLearn 2024 abstract submission deadline has been extended to Aug 5, 2024. If you are working on cutting-edge ML research, please consider submitting an abstract. BayLearn 2024 will be an in-person event, held on Oct 10, 2024: https://baylearn.org This is a great opportunity for student researchers to showcase their latest research work and get good feedback. Submissions are non-archival and the review process is quite informal. #machinelearning #research

68

April was an exciting month for AI at Meta! We launched MTIA v2 https://lnkd.in/gvC-UgVN, Llama3 https://lnkd.in/gdbfXa-6, presented a tutorial and paper on the PyTorch2 compiler at ASPLOS https://lnkd.in/gsT9DUr5, released PyTorch 2.3 https://lnkd.in/gYn3Vnzm and, to top it off, we launched the PyTorch ecosystem solution for mobile and edge deployments, ExecuTorch Alpha https://lnkd.in/gWv96jpm optimized for Large Language Models. What better than to combine all of these... running Llama3 on an a mobile phone exported with the PT2 Compiler's torch.export, and optimized for mobile deployment. And you can do all of this in an easy-to-use self-service format starting today, for both iPhone and Android as well as many other mobile/edge devices. The video below shows Llama3 running on an iPhone. (Makers will love how well models run on Raspberry Pi 5!) PyTorch is being developed by a multi-disciplinary team comprising ML engineers, accelerator experts, compiler developers, hardware architects, chip designers, HPC developers, mobile developers, and specialists and generalists that are comfortable across many of the layers involved in building end-to-end solutions. Even better -- if you're excited by the possibilities of AI, and solving the system design challenges of making AI run well across all hardware types, we are looking for YOU! The Pytorch team has openings across PyTorch core, compilers, accelerators and HW/SW co-design https://lnkd.in/gu2Y_KpY https://lnkd.in/gQVXBP6P and a broad range of positions that involve PyTorch from model development all the way to hardware deployments https://lnkd.in/gjDdUh5w #PyTorch #ExecuTorch #Llama3 #AICompilers #MTIA #AcceleratedAI #MetaAI #Meta

64

I’m thrilled to announce that I’ll be serving as a Chair of the NAACL 2025 Industry Track, and our Call for Papers is now open! https://lnkd.in/gBM2F4Fv If you're working on real-world NLP applications, I highly encourage you to submit your work. With the rapid development observed recently in Generative Ai and large language models (LLM), this year we are particularly encouraging submissions of papers leveraging this technology to develop real-world applications. Potential topics include, but are not limited to: - Best practices and lessons learned - Continuous maintenance and improvement of deployed systems - Case studies from design to deployment - Design of application-relevant datasets - Ethics and bias in deployed NLP systems - Handling unexpected user behavior - Implementation at speed, scale, and low-cost - Negative results related to real-world applications - Novel previously unsolved NLP problems - Offline and online system evaluation methodologies - Online learning for deployed systems - Robustness against various perturbations in deployed systems - System combination and hybridization In addition, opinion/vision papers related to real-world applications are also welcome! The conference will be held in Albuquerque #naacl2025 #industrytrack #nlproc

67

3 Comments

Crafting effective prompts is an art itself, but getting it right can mean the difference between a great output and total garbage! Today, we’ll explore the 5 prompt engineering techniques that every AI engineer should know to get the most out of LLMs. #llms #llmops #chatgpt4 #rag #gemini #ai https://lnkd.in/d3an85KE

3

Need tips on how to accelerate deep learning training? Check out this blog post #deeplearning #matlab

1

Talking about LMMs and Astrophysics soon! :)

7

I believe we are in endgames of newer models development, and these are saturating fast. The next frontier will be optimizing inference infrastructure cost. Basically with #openai #o1 models, we can see that the value chain is quickly shifting to the right side from pre training, post training to inferencing. In simple words , the most important task while putting on the thinking cap and getting on the train of thoughts is to know "when to stop". Hence models like #openAI #o1 will be hard to scale without optimising inferencing. Ok, I should stop my chain of thoughts here before I exhaust all my inferencing capacity. #strawberry #ml #ai #genAI

4

Agentic AI is revolutionizing the industry of legacy code migration and modernization where we see clear positive business and societal impacts of this disruptive technology.

6