Bowen Liu

Introducing RouteLLM: a sophisticated routing framework developed with Berkeley-LMSys in collaboration with Anyscale. RouteLLM optimizes query handling by dynamically selecting between high-performance proprietary LLMs and cost-effective open-source models, cutting costs by over 2x without sacrificing quality. Using human preference data and LLM-as-a-judge for data augmentation, our routers evaluate query complexity to choose the appropriate model. Rigorous testing on benchmarks like MMLU and GSM8K confirms our cost-efficient, high-quality performance. Explore our open-source code, models, and preference data on GitHub: https://lnkd.in/gJyijZRx, and try our online demo: https://lnkd.in/gME-GC2a Explore More about RouteLLM and learn how it can revolutionize your LLM applications: Read our Blog here: https://lnkd.in/gfPD-u-y LLMsys Blog here: https://lnkd.in/ga_MgERE Full research paper here: https://lnkd.in/gqRy7Pjy

81

1 Comment

🚀 Looking to streamline fine-tuning for models like Llama 3, Mistral, and Mixtral? Our latest blog shows how Anyscale simplifies LLMs fine-tuning with Ray: 🔑 Key takeaways: • Scalable fine-tuning with LLM-Forge • Model Deployment with RayLLM Read more: https://lnkd.in/gQwD4A2U

34

Do #LLMs really lack common sense? Is it “able to solve Olympiad problems but fails at basic math?” https://lnkd.in/gvWdw8Kq Well, not all LLMs are poor at basic math! The #mathstral model from Mistral AI does a good job. With LlamaEdge, it runs effortlessly on my laptop. https://lnkd.in/gt7h5Ghy

3

Today, we’re launching Substrate. We’re also announcing our $8M Series Seed led by Lightspeed. We believe the most robust, reliable, and productive uses of AI are when many models are used in coordination with each other, with regular programming mixed in. Most people building with AI already know this. But the available tools are suboptimal, unless you work for Google. For the rest of us, we’re left creating an unwieldy mess of chained API calls to multiple providers, with many roundtrips in between. We took a hard look at this state of affairs, and recognized how much it is stifling progress. Building large multi-step AI workloads requires sophisticated high-performance tooling and infrastructure. Nobody wants to deal with more tooling and infrastructure… but everyone would benefit from simple, intuitive interfaces that abstract away a powerful system underneath. No tooling, no infrastructure – just elegant abstractions.

27

1 Comment

The best part of investing at the earliest stages of company building is every once in a while you meet founders who can see the future. Ben Guo & Rob Cheung are two of those founders and after a year of building in stealth they're launching ꩜Substrate, the first inference API optimized for multi-step AI workloads. 👉 https://substrate.run With ꩜Substrate, you connect nodes from a curated library that includes optimized ML models, built-in file and vector storage, a code interpreter, and logical control flow. By simply connecting nodes, you describe a graph workflow, which Substrate then analyzes and runs as fast as possible. Entire graphs of many nodes will often run on a single machine, with microsecond communication between tasks. We are thrilled to participate in their 💵 $8M Seed led by Lightspeed (Guru Chahal/Nnamdi Iregbulem) alongside our friends at South Park Commons (Arian Agrawal), Craft Ventures, Red Swan Ventures (Eric Rubin/Sandy Cass), joined by amazing 🧠 founders and 👨‍🏭 operators like Guillermo Rauch, Immad Akhund, William Gaybrick, Chris Best, Gokul Rajaram, Shreyas Doshi, Jana Messerschmidt, Michael Manapat, and Matthew Hartman.

23

1 Comment

Combining the power of TensorFlow, NeuralForecast, and zero-shot LLMs, Jinhang Jiang shows how we can generate accurate multi-step quantile forecasts.

11

Label Studio Customized Backend for Semiautomatic Image Segmentation Labeling - Customized backend; GCP Deployment; Data Versioning with GCS Integration by Yuhan Yao

3

Check out our recent blog post collaboration: Building a RAG Batch Inference Pipeline with Anyscale and Union.ai! If you have any questions find some of us this week at #RaySummit! 🔗 https://lnkd.in/gjDKRPmu

7

In the lead-up to #Raysummit, we have a guest blog on Building a RAG Batch Inference Pipeline with Anyscale and Union.ai 🚀 #Flyte is an open-source orchestrator facilitating production-grade data & ML pipelines. When combined with Ray’s distributed computing power, it unites data, platform, infrastructure, and ML engineers to boost productivity and scale across evolving use cases. Union.ai platform makes orchestrating these pipelines seamless. In the RAG Batch Inference pipeline exactly there are actually 2 pipelines that Union orchestrates: 🔹 Embedding Generation Pipeline 🔹 Batch Inference Pipeline #Flyte Deck on Union.ai lets you preview responses without downloading, ensuring instant validation. Anyscale’s Ray platform optimizes the execution of these pipelines, helping deliver leading performance and cost efficiency. Read more here 👉 https://lnkd.in/gjDKRPmu

47

🌐 Exciting Announcement! We are thrilled to host Fangru Lin for a discussion on her groundbreaking work, "Graph-enhanced Large Language Models in Asynchronous Plan Reasoning," recently presented at ICML 2024! 🎉🧠 Join us on August 24, 2024, at 10:00 AM (EST) as Fangru Lin walks us through the innovative "Plan Like a Graph (PLaG)" approach. This method revolutionizes LLM performance by breaking down tasks into sub-tasks, forming an execution graph that is either solved in parallel or sequentially by the LLM. This structured approach not only enhances results but also opens new pathways for future research in prompt engineering. 📜 Paper Abstract: Planning is a fundamental property of human intelligence. Reasoning about asynchronous plans is challenging since it requires sequential and parallel planning to optimize time costs. Can large language models (LLMs) succeed at this task? Here, we present the first large-scale study investigating this question. We find that a representative set of closed and open-source LLMs, including GPT-4 and LLaMA-2, behave poorly when not supplied with illustrations about the task-solving process in our benchmark AsyncHow. We propose a novel technique called Plan Like a Graph (PLaG) that combines graphs with natural language prompts and achieves state-of-the-art results. We show that although PLaG can boost model performance, LLMs still suffer from drastic degradation when task complexity increases, highlighting the limits of utilizing LLMs for simulating digital devices. Our study is an exciting step towards using LLMs as efficient autonomous agents. Our code and data are available at https://lnkd.in/e49YXamU. Check out the full paper: https://lnkd.in/eCdsdhE4. 📅 Save the Date: 🗓️ August 24, 2024 🕙 10:00 AM (EST) 📍 Location: 🌐 Virtual Event Join via Zoom: https://lnkd.in/eSq8PKNe 🚀 Don't miss this opportunity to gain insights into the future of LLMs and their application in complex tasks! 🌟 #ICML2024 #LLMs #MachineLearning #GraphLearning #AIResearch Looking forward to your participation! Spread the word and let's dive into the future of LLMs together! 🚀

13

LanceDB Raises $8M for AI-Optimized Vector Database - LanceDB raises $8M seed round: CRV led the investment, with participation from Essence VC and Swift Ventures, bringing total funding to $11M. - AI model optimization with LanceDB: The open-source database stores billions of vectors and raw data, enhancing AI application performance up to 100 times faster than Parquet. - Launch of LanceDB Cloud and Enterprise: The seed funding will help launch LanceDB Cloud and Enterprise, offering managed services and enhanced cybersecurity features. Subscribe to our daily newsletter here for more AI news https://lnkd.in/dW_9hGsj #ArtificialIntelligence #Innovation #BigData #DataScience #Analytics #Technology #Startups #VentureCapital #CloudComputing #Entrepreneurship #AI

2

Visionaries like Ion Stoica are always starting something - like Opaque, Anyscale, Databricks and Conviva, the 4 groundbreaking companies he co-founded. As Director of University of California, Berkeley’s Sky Lab, he’s dreaming up even more - so we invited him and his brilliant mind to shed light on the future of #trustworthyAI at our upcoming Confidential Computing Summit. Find out more: https://hubs.la/Q02rpDrf0 Join world-renowned academics like Ion, as well as #confidentialcomputing leaders and industry decision-makers in security, data, and IT, as we gather in San Francisco to solve the puzzle of #confidentialdata and unlock #privacypreservingAI for the enterprise. This is the year of confidential computing - be there to help us crack the code by reserving your spot now 👉 Register now: https://hubs.la/Q02rpy8Z0 #CCSummit #buildingtrustworthyAI

14

Introducing a new short course: Function-Calling and Data Extraction with LLMs! Made in collaboration with Nexusflow and taught by Jiantao Jiao, Co-founder & CEO, and Venkat Srinivasan, Founding Engineer, this course focuses on two key skills for building LLM applications: function-calling and structured data extraction. Function-calling allows LLMs to execute external functions based on natural language instructions, while structured data extraction enables LLMs to retrieve useful information from unstructured text. 🛠️ Learn how function-calling is used 🤖 Use an LLM with multiple function calls 🌐 Build function calls that can access web services 📊 Use function-calling to extract structured data And more. Sign up for free: https://hubs.la/Q02CDGGZ0

743

19 Comments

In the cutting-edge realm of AI and machine learning, fine-tuning the performance of large language models (LLMs) like Llama-3 remains a core challenge. The recent study by Junsang Yoon, Akshat Gupta, and Gopala Anumanchipalli from UC Berkeley provides a fascinating look into the efficacy of model editing techniques. Why This Paper is Crucial: ➡️ Innovative Techniques: The paper meticulously analyzes various model editing strategies such as ROME, MEMIT, and EMMET, exploring their effectiveness in achieving precise model improvements without comprehensive retraining. ➡️ Optimal Batch Size Analysis: One of the key findings is that larger edit batch sizes might not always yield the best outcomes. Instead, smaller, more frequent sequential edits can enhance model performance more effectively, challenging the prevailing notion that bigger is better. Key Insights: ➡️ Sequential vs. Batch Editing: The research indicates that sequential model editing, which involves making smaller batch size edits over time, often results in better overall model performance compared to larger single batch edits. ➡️ Future of Model Editing: This paper highlights the potential limitations of current model editing methods that prioritize larger batches, paving the way for future investigations into optimizing batch sizes for better performance. #analyticsvidhya #datascience #generativeai

18

From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, Huaming Chen University of Sydney 2024 https://lnkd.in/gVZ3r8G7 "As LLMs have demonstrated significant capabilities in tasks such as code generation and vulnerability detection, researchers are increasingly interested in their potential applications. However, the study highlights the limitations of LLMs, including their lack of autonomy and self-improvement, which LLM-based agents aim to address. These agents represent a promising advancement towards achieving Artificial General Intelligence (AGI) by integrating LLMs for decision-making and action-taking. Despite the growing body of research on LLMs in software engineering, there remains a lack of clarity distinguishing between LLMs and LLM-based agents, as well as a need for standardized benchmarks to evaluate these technologies. The survey systematically investigates current practices and solutions related to both LLMs and LLM-based agents, focusing on six key areas: requirement engineering, code generation, autonomous decision-making, software design, test generation, and software maintenance. By reviewing and contrasting the contributions of LLMs and LLM-based agents across these topics, the survey provides insights into their respective tasks, benchmarks, and evaluation metrics. It also discusses the models and benchmarks utilized in this domain, offering a comprehensive analysis of their effectiveness and applications in software engineering. Ultimately, the authors aim to illuminate the path forward for LLM-based agents in software engineering, paving the way for future research and development in this promising area."

2

For his debut TDS article, Jeremi Nuer presents a thoughtful walkthrough of a recent paper on model interpretability: "We want to have finer-tuned control over LLMs. Being able to detect when certain features are active, and tune how active they are in the middle of generating output, is an amazing tool to have in our toolbox."

18

Fine-tuning LLMs with Reinforcement Learning from Human Feedback This blog post explores how Reinforcement Learning from Human Feedback (RLHF) can be used to fine-tune Large Language Models (LLMs) to align with human preferences. It delves into the concept of Proximal Policy Optimization (PPO) and how it is used in this context. The key takeaway is that RLHF offers a powerful framework to train LLMs that can produce human-aligned text, while still maintaining creativity. This paves the way for more natural and nuanced language generation in the future of AI and natural language understanding. Read the article: https://lnkd.in/d9EGf579 #AI #machinelearning #goML

8