Xiujia Guo

Blog Post: https://lnkd.in/d__xrtJA > Retrieval-augmented generation (RAG) systems use many different algorithms to chunk, embed, and rank textual content for enabling natural language based question-answering on private knowledge bases. In a recent article, Anthropic introduced Contextual Retrieval, a new chunking algorithm that surpasses state-of-the-art (SOTA) methods and released the “Codebases dataset” for its benchmarking. In this article, we evaluate Yurts’ RAG pipeline on this new benchmarking dataset. This evaluation highlights that Yurts’ RAG system matches the performance of Contextual Retrieval, while operating at only 1/300th of the cost, underscoring our commitment to providing high performance solutions that are also cost-effective for our customers. Through our analysis, we also highlight some key challenges that arise when using new benchmarking datasets to evaluate RAG platforms, offering insights for those navigating this evolving space.

7

1 Comment

🚀 Amazon S3 (Simple Storage Service) achieves its high durability, quantified as an astonishing 11 nines(99.999999999%) of durability. Here’s how they do it: 🔸 Data Redundancy: Utilizing mechanical hard disks for their cost-effective storage, Amazon S3 replicates data across multiple disks, employing erasure coding to split data into shards, significantly reducing the probability of data loss. 🔸 Data Integrity: To combat bit flips and data corruption, Amazon S3 implements checksums using client SDKs, ensuring data integrity through rigorous checks and the use of efficient checksum algorithms. 🔸 Data Audit: Amazon S3 periodically scans storage devices to verify data integrity, quickly re-replicating data if any issues are detected, maintaining a balance between repair and failure rates. 🔸 Data Isolation: By storing metadata and file content separately and operating isolated data centers within an availability zone, Amazon S3 minimizes the risk of simultaneous failures. The commitment to data durability is evident in every layer of Amazon S3’s design, from the hardware to the engineering culture. It’s a testament to the power of thoughtful system design and its impact on reliability. #AmazonS3 #DataDurability #SystemDesign #CloudStorage #DataIntegrity #EngineeringCulture

5

I had a great time guest lecturing at UC Berkeley about how to use LLMs from Python: https://lnkd.in/gjHmRzdG Questions I got after from students: - how do I get internships in llms? - how to get into ml research? Do you have to be a math genius? - how does the attention mechanism work, and how does it relate to Cog Science brain frameworks? - can you do fine-tuning on a mac? - how do optimize the retrieval step of RAG for maximum relevance? - in what way does parameter size affect the answer quality? I also met Leanid Palkhouski who’s working on a startup around GPU rental. Very cool!

60

3 Comments

Exciting news for anyone interested in building their own Large Language Model (LLM) from the ground up! 📘 If you're ready to dive into the process and deepen your understanding of the technologies behind LLMs, I highly recommend checking out this book: "Build Your Own Large Language Model from Scratch". This book is perfect for those who already have a foundation in programming, especially in Python. It guides you step-by-step through the intricate concepts, from preprocessing to model training, that are essential in creating an LLM. For me, it was both an enlightening read and a hands-on experience that boosted my understanding of this cutting-edge field. If you're ready to take your skills to the next level and create your own LLM, don’t miss out on this resource!

17

3 Comments

Fantastic news today on the general availability of Amazon Q. Amazon Q is the most capable generative AI-powered assistant for accelerating software development and leveraging companies’ internal data. It eliminates tedious work for developers and employees across organizations. #AmazonQ #GenerativeAI

15

Just got back from the AI Engineer World's Fair in SF, and wow, what a ride! 🚀 3 days, 2000 AI nerds, and more mind-blowing ideas than I could count. For once, I didn't have to explain what I do for a living – everyone just got it. The event was packed with insightful presentations, hands-on workshops, and cutting-edge demos that truly showcased the future of AI. Industry experts from Google, Perplexity, Discord, Tinder, and Zapier shared their experiences and insights into building AI products. Key takeaways that are still buzzing in my mind: 1. Evaluation is everything: What can't be measured, can't be optimized. 2. RAG (Retrieval-Augmented Generation) is more than just vector embeddings. Hybrid, modular, and graph retrievers (neo4j) are the new hotness for handling complex scenarios. 3. Inference speed is getting cheaper and faster. Startups like Fireworks AI, @Grokk are pushing the boundaries here. 4. Co-Pilots are here to stay. Companies across finance, legal, insurance, and storage are all trying to integrate co-pilots into existing applications. 5. Engineers with product sense will have an edge in this rapidly evolving field. But the real magic? Those unplanned hallway chats. I met startup founders, working on Agents/Memory/Inference, geeked out with industry counterparts about NL2SQL, Evals, Generative UI, and nearly missed a session brainstorming with a fellow hacker. Vinay Pinnaka Anil Krishna C. Matthew Ambrogi Eugene Yan Abhinav Gupta Patrick Debois John Knox Jayasimhan Masilamani The conference was a stark reminder that the field is moving at lightspeed, and it's clear that continuous learning is the name of the game. I will most definitely be back to soak in more of this vibrant atmosphere next year. Until then, I am looking forward to applying the insights I picked up this year. https://lnkd.in/ge5A5eXS #AIWorldFair #SFTech

38

5 Comments

I'm happy to share my latest blog post on Medium about fine-tuning the Phi3-mini-4k-instruct model on custom data and running CPU inference. This guide covers Small Language Models, using Unsloth for efficient fine-tuning, and leveraging ONNX for optimized inference. Check it out if you're interested in making AI more accessible and efficient. https://lnkd.in/eaQ-jVXy

23

2 Comments

I'm excited to share this article that Melanie Chen and I wrote on feature stores for Chalk's new blog! In this piece we take a deep architectural dive into feature stores—a critical framework for building production ML systems. https://lnkd.in/gxWNzGxE

19

April was an exciting month for AI at Meta! We launched MTIA v2 https://lnkd.in/gvC-UgVN, Llama3 https://lnkd.in/gdbfXa-6, presented a tutorial and paper on the PyTorch2 compiler at ASPLOS https://lnkd.in/gsT9DUr5, released PyTorch 2.3 https://lnkd.in/gYn3Vnzm and, to top it off, we launched the PyTorch ecosystem solution for mobile and edge deployments, ExecuTorch Alpha https://lnkd.in/gWv96jpm optimized for Large Language Models. What better than to combine all of these... running Llama3 on an a mobile phone exported with the PT2 Compiler's torch.export, and optimized for mobile deployment. And you can do all of this in an easy-to-use self-service format starting today, for both iPhone and Android as well as many other mobile/edge devices. The video below shows Llama3 running on an iPhone. (Makers will love how well models run on Raspberry Pi 5!) PyTorch is being developed by a multi-disciplinary team comprising ML engineers, accelerator experts, compiler developers, hardware architects, chip designers, HPC developers, mobile developers, and specialists and generalists that are comfortable across many of the layers involved in building end-to-end solutions. Even better -- if you're excited by the possibilities of AI, and solving the system design challenges of making AI run well across all hardware types, we are looking for YOU! The Pytorch team has openings across PyTorch core, compilers, accelerators and HW/SW co-design https://lnkd.in/gu2Y_KpY https://lnkd.in/gQVXBP6P and a broad range of positions that involve PyTorch from model development all the way to hardware deployments https://lnkd.in/gjDdUh5w #PyTorch #ExecuTorch #Llama3 #AICompilers #MTIA #AcceleratedAI #MetaAI #Meta

64

A recent breakthrough titled "Matrix Multiplication-Free LLMs" demonstrates a huge advancement in the area of Large Language Models (LLMs) by reducing computational costs. The authors have eliminated MatMul operations from LLMs, claiming to 10 times reduction in memory usage and a 25.6% increase in training speed, all while maintaining strong performance at billion-parameter scales. Paper link: https://lnkd.in/ggph8qXc #AI #machinelearning #deeplearning #LLMs

16

Very useful LLM development tool

2

I've been considering the potential of large language models (LLMs) in the realm of therapy. It's arguable that LLMs can sometimes provide valuable insights that human therapists might miss, given the vast amount of data they are trained on, which allows them to generate thoughtful suggestions. However, their inability to think and reason with unseen data points remains a significant obstacle to fully autonomous AI therapy. What if the first output token is incorrect? The final output can diverge due to the NTP strategy. Could we fine-tune these models with multi-modal inputs—such as speech, video, and text—to better mimic real-world scenarios and assist human therapists?

12

2 Comments

🚀 New Medium Article Alert! 🚀 I’ve just published an article where I explore Overview of LLMs, GPT Models, RAG, and LlamaIndex Read it now 👉 https://lnkd.in/gARHY54f #AI #AZURE #LLM #RAG #GPT #Tech #Programming #Medium

13

Excited to share our latest work on multimodal pre-training MoMa! When working with early fusion mixed modal models (natively multimodal  in/out), Chameleon showed the text LLM architecture (e.g. Llama) doesn’t scale to early fusion. While Chameleon proved the significant quality and efficiency benefits of early fusion models, there is a core issue with the inherent differences in information across text and image tokens being treated uniformly. We propose MoMa (Mixture of Modality-aware experts), a novel adaptive architecture that explores adaptivity across modality, width, and depth, delivering up to 4x FLOPs savings (4x smaller or 4x cheaper for the same quality)! Building on the success of MoE in Text LLMs (e.g. Mixtral, Deepseek-MoE,  Grok), MoMa leverages a MoE (mixture-of-experts) block for each Modality showing for early fusion LLMs this is a much more efficient approach compared to dense or MoE alternatives. We further show empirical scaling of MoMa, extensions with MoD (mixture-of-depths), and Upcycling to propose our core training recipe. MoMa is our first step in re-thinking the core architecture primitives when building multimodal early fusion LLMs, we believe there’s a lot more potential in further exploring adaptive compute. This is joint work with amazing co-first authors Victoria Lin, Armen Aghajanyan and co-authors – Liang Luo, Srini Iyer, Mike Lewis, Gargi Ghosh and Luke Zettlemoyer. Paper: https://lnkd.in/gvKBj8SQ Twitter: https://lnkd.in/gj8TG7iR

235

1 Comment

https://lnkd.in/gRenYMXW I really enjoy this distributed training guide from Lambda Labs. It starts from a single gpu to train a moderate GPT2 model, and progressively evolve code and settings to utilize multi-node multi-gpu, all the way up to training Llama 405B. With my learning budget, I likely will stay under 7B models for a while. But these distributed training recipes surely help me run experiments faster.

16

3 Comments

Thanks for making a podcast on this blog! This blog is indeed a valuable asset to the recommender system industry. I don't see much sharing on recommender system monitoring. Kudos to the Tubi team!

31

I'm seeing demand for two kinds of workshops in my work at the moment. One is enablement-style workshops like this one where the idea is to enable folks in AI technologies - (usually) tech-strong but new to AI and wanting to benefit from increasingly accessible technologies. The second is more where folks want help aligning the possibilities of AI with their business goals, choosing or developing use-cases, and designing solutions incorporating AI. Room for both, often as part of the same journey.

14

1 Comment

I’m excited to share news of our upcoming Coinbase ML and Blockchain Summit, where Prof. Salman Avestimehr, USC Viterbi School of Engineering Professor and CEO of FEDML, will be speaking. FEDML is making groundbreaking strides in decentralized AI, addressing the challenges of computational demand and data management in large language models (LLMs). Decentralization not only makes sense for individual LLMs, which serve as standalone services, but it also enhances their value relative to operational costs. Imagine a world where everyone can contribute to and benefit from data and computing power—this could truly revolutionize AI. Join us on May 2nd to explore the potential of decentralized AI with a pioneer in the field. Registration link in thread. #blockchain #ai #ml #coinbase #FEDML

16

1 Comment