Summer Yue

I'm excited to share two papers from the Retrieval team at Walmart that were presented at CIKM last week. They're both about improving the relevance of Embedding-Based Retrieval. 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲 𝗙𝗶𝗹𝘁𝗲𝗿𝗶𝗻𝗴 𝗳𝗼𝗿 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴-𝗯𝗮𝘀𝗲𝗱 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹. https://lnkd.in/gNQe7QhA The first is about how to intelligently choose a query-dependent cutoff on the cosine score to remove irrelevant items from the retrieved set. Since cosine scores aren't comparable across queries, we introduce a "Cosine Adapter" to map the scores to interpretable scores. 𝗘𝗻𝗵𝗮𝗻𝗰𝗶𝗻𝗴 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲 𝗼𝗳 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴-𝗯𝗮𝘀𝗲𝗱 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗮𝘁 𝗪𝗮𝗹𝗺𝗮𝗿𝘁. https://lnkd.in/gdqkc2hR The second is about model training techniques we learned to improve the relevance of embedding models. In particular, we describe how to use a teacher model to provide relevance labels for training Juexin (Joy) Lin Nicholas Rossi, PhD Feng LIU Zhen Yang Sachin Yadav Praveen Reddy Suram Satya Chembolu Prijith Chandran Hrushikesh Mohapatra Alessandro Magnani Ciya Liao

139

1 Comment

Hey, I'd like to give a shout-out to ClearML for helping us at Vimeo develop AI-based systems. We have been happy customers for a few years now and I'd like to share a bit about how we use ClearML. We use ClearML Datasets to cache a dataset of video transcripts, and run testing loading directly from ClearML Datasets. This allows us to simultaneously speed up the data loading and ensure consistency. Moreover we use ClearML to save our benchmark annotations in one place and track our system performance on the benchmark every-time we run an experiment or change our prompts. ClearML allows us to see the different parameters and prompts that were used for each experiment and to monitor improvements or regressions in our performance. We also use ClearML to run large-scale tests and help with statistical evaluation of our methods. For example, we developed a RAG (Retrieval Augmented Generation) Q&A system and wanted to verify that the LLM will not answer certain questions or user queries that are outside the scope of the video. We used ClearML to collect and analyze the RAG responses on many videos for predefined user queries that were outside the scope of the videos and got good visibility into the performance of the system. Also, the comparison feature on ClearML is great for tracking the improvement of our metrics along the progressing versions of our systems.

32

1 Comment

I just published a blog on how to fine tune LLMs and create custom datasets. In this blog, I cover: - What fine-tuning is and its benefits - When to fine-tune a model - Methods for fine-tuning, including full fine-tuning and parameter-efficient fine-tuning - How to prepare your custom dataset Check it out here: https://lnkd.in/gDcrrBGP

15

1 Comment

Really excited about KIP-932 (Queues for #Kafka) and its impact on the AI world: massive worker architectures to index unstructured documents at scale, feeding LLMs. Combined to inference cost reduction, lower latency, and smaller specialized models: local #SLMs will be a game-changer for scaling AI workloads cost-effectively in the streaming paradigm. (SLM: Small Language Model)

57

1 Comment

Day 3/3 GenAI Summit San Francisco Causal AI, causal grounded RAG - cRAG, scaling from POC to production in AWS, a deep dive into the evolution of Llama models and what's coming next, synthetic data in the age of genAI, PII, and domain randomization. I can't count how many times I've heard the words AI, genAI, data, cloud, RAG, LLM, and multimodal in the last 3 intense days. And when I finally went to have lunch at a restaurant, at the adjacent table, a fintech AI startup founder was pitching to a VC, while the people at the table behind me were talking about data engineering. It feels so good to be in SF, where at the airport’s security check-in a big Anthropic advertisement greets you! 😊 The last 3 days were amazing, filled with inspiration and thought-provoking new tech. I can't wait to get back to building DiffuseDrive and working with our customers!

22

⚡ New blog post about our work on hybrid retrieval for #Instacart search While it's widely recognized that adopting a hybrid retrieval system (keyword + vector) can enhance performance, creating a bespoke solution tailored to our specific business needs proved to be quite a challenge! In our latest blog post, we explore: 1. The current search retrieval architecture at Instacart. 2. The challenges we faced, including the variance in search query distribution and the number of relevant documents per query. 3. Our methods for contextually adapting the recall set from each retrieval source to boost the precision and recall of the final results. Super proud of the team who made this possible Vinesh Gudla Prakash Reddy Putta Ankit mittal Andrew Tanner Guanhua S. Xiao Xiao Taesik Na Alex Charlton Xukai Tang Akshay Nair Haixun Wang

52

A new tool for evaluating LLM. This could be useful for enterprises.

7

Replacing an expensive generalist LLM like GPT-4 with a cheaper more specialized model like a fine-tuned Mistral 7B is a tried and true approach to really drive your GenAI expenses down. We've got a handy guide on how to start your fine LLM cost optimization journey with OpenPipe for fine tuning and OctoAI for model serving https://lnkd.in/e8udh8de

84

Why are data scientists and ML people so excited by the DSPy library? If you come from an academic background or from "traditional" NLP, you probably find the whole LLM app building process quite manual and non-principled. Everything before was about running experiments, making sure you wouldn't overfit on your training data, and doing model parameter optimisation. Now with LLMs a large part of the model is encoded in a prompt, but prompt engineering is more black art than science. DSPy is a framework that tries to automate that by automatically picking the best examples to add to your prompt. We went ahead and tested this with Haystack, and it works! My colleague Stefano Fiorucci created a notebook where we show how to optimise a prompt in a RAG pipeline to make your LLM less verbose while keeping the accuracy high: https://lnkd.in/d9NGnMwR I wrote a post about that same notebook in medium with a short introduction to DSPy: https://lnkd.in/dTs8mMNs

52

1 Comment

If you are a data scientist looking to work on the hardest challenges at the edge of implementing AI models in the enterprise. If you believe that AI Agents will make us 100X more productive at our work. If you want to live in a future where an Agent can go through all of your main KPIs and provide you a summary every morning with suggested key actions. If you are convinced that companies should have ultimate freedom and control over the agents and models they deploy - without having to deal with the technical complexities this entails. If you understand that the only way to make this happen lies in helping companies integrate fragmented data, create a semantic layer, and effectively orchestrate LLMs. If you want to work with exceptional leaders like Raul Mosqueda - and you understand the deep technical achievements presented in his demo: Please reach out. Arkham is your home.

19

1 Comment

I just published Vol. 67 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 A Comprehensive Study of Noise Types in RAG, from Wu et al. 📚 Lessons from Adobe Express’s Iterative Enhancement of Multi-Modal Search, from Adobe 📚 Dynamic Hard Negative Mining for Enhanced Text Embeddings, from Tencent 📚 Tackling the Unique Challenges of Location-Based Search, from Airbnb 📚 Jina-ColBERT-v2: Enhancing ColBERT for Multilingual Retrieval, from Jha et al. 📚 Enhancing Music Recommendation through Psychology-Informed Session Embeddings, from Deezer 📚 Analyzing [MASK] Token Behavior in ColBERTv2, from RIT 📚 A Critical Evaluation of Traditional Models vs. LLMs in Sequential Recommendation, from Xu et al. 📚 Collaborative Discourse for Complex Information Discovery, from Jiang et al. 📚 A Large-Scale Synthetic Test Collection for Advancing Passage Retrieval Research, from Rahmani et al. #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

34

1 Comment

AI at Meta 🤝 Roboflow: Open source computer vision New post on the Meta AI blog: how Segment Anything has saved over 74 𝐲𝐞𝐚𝐫𝐬 (and counting) for hundreds of thousands of users curating computer vision datasets on Roboflow. Amazingly, even this under estimates SAM's measured impact by an order of magnitude, i.e. all our open source work can be self hosted: using SAM in Workflows for chaining together models in inference, running SAM on edge devices, and with autodistill for local distillation. Link in comments ↓ #opensource #computervision #segmentanything #AI

262

13 Comments

"To tackle this issue, we developed a Whole Page Ranker model which determines the optimal positions for the new content on the page. The objective of the model is to ensure that we show highly relevant content to the users while also balancing revenue objectives. By dynamically adjusting the layout based on the content type and relevance, we could also present information more intuitively." #computationalinteraction #computationaldesign

1

Can LLMs Avoid Costly Matrix Multiplication Overhead? I recently read an interesting paper that describes how to eliminate matrix multiplication (MatMul) from LLM training and inference processes. Scalable MatMul-free Language Modeling (https://lnkd.in/gpfxnKr3) compares standard self attention and MatMul-free alternatives. Reducing or eliminating the overhead of matrix multiplication will have a major impact on the cost of building and using AI. This also shows there is substantial room for improvement over pure GenAI solutions. I believe that the components for actual AGI are starting to be revealed. It could be some time before actual AGI emerges, but incredibly useful AI has already been created and is being deployed in controlled situations. The foundation model efforts to find AGI have provided powerful tools for coding and image/video production that can be immediately applied to an ever wider range of tasks. The discovery that these models can be made more cost effective will increase the number of implementations that will be possible.

11

1 Comment

I just published Vol. 59 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Unifying Context Ranking and Generation in Retrieval-Augmented LLMs, from NVIDIA 📚 A Systematic Approach to Identifying Best Practices in Retrieval-Augmented Generation, from Fudan University 📚 Exploring the Mutual Benefits of LLMs and Search Engine Integration, from IEEE Members 📚 Rethinking Ranking Algorithms for Map Interfaces in Airbnb Search, from Airbnb 📚 Reassessing Context Relevance in LLM Question Answering, from USC 📚 A Benchmark for Evaluating Long-Context Language Models and RAG Systems, from Salesforce AI Research 📚 A New Taxonomy for Long-Context Language Model Evaluation, from Bar-Ilan University 📚 A Multi-Stage Content-Centric Framework for Advancing Retrieval-Augmented Generation, from Tsinghua University 📚 Efficient Document Expansion for E-commerce Search, from Walmart 📚 A Comparative Study of RAG vs. Fine-Tuning in API-Driven Code Generation, from Microsoft #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

67

2 Comments

(MatMul) typically dominates the overall computational cost of large language models (In general, AB ≠ BA), fun huh? As you can imagine, this cost only grows as LLMs scale to larger embedding dimensions and context lengths. In this work, they show that MatMul operations can be completely eliminated from LLMs while maintaining strong performance at billion-parameter scales. Removing MatMul from the calculations in large language models means these models don't need powerful computers to run. This change allows them to work on simpler devices, like smaller servers or even some personal computers, making advanced AI tools available to more people and places. Hardware Optimization: Custom accelerators demonstrate the practicality of this method by processing billion-parameter models with just 13 watts of power. #whynividia

6

From my conversations, I'm noticing different trends between how AI engineers deploy evaluation models (model as a judge) in production monitoring vs application testing. 💡 Value - Monitoring: provides real time alerts of performance or used to preemptively block problematic outputs from surfacing to the user -Testing: regression testing applications to establish baseline performance before pushing to deployment ⚡ Speed - Monitoring: near real time, requires extremely fast token processing - Testing: batch, async, often processing large test sets, lower token processing speed required 🎯 Accuracy - Monitoring: overindex on false positives, better to flag something as problematic and be wrong than to let something slip between the cracks - Testing: high correlation with human graders, be as accurate and robust as possible 💰 Cost - Monitoring: prefer cheaper models, otherwise the additional inference cost per generation will add up exponentially - Testing: willing to take on higher costs, calculation is accurate testing leads to better performing outputs 🛑 Risk - Monitoring: if your monitoring judge gets something wrong, there is a high consequence because it directly impacts the end user - Testing: if your testing judge gets something wrong, async review gives some potential to remediate the issue At the moment, it seems using model as a judge for testing is a little more popular due to the high cost constraints of constant monitoring, but that is subject to change as more specialized models come out. As the industry matures, I strongly believe we're gonna see specialization between models for monitoring and models for testing. If you're currently using model as a judge techniques, would love to chat more! Let me know what you think. Twitter Handle: https://lnkd.in/eAsQvtiu

26

That's a good summary Manny Bernabe - Reflecting on an insightful article from O'Reilly about LLMs, I was struck by this point: "LLMs are also broadly accessible, allowing everyone, not just ML engineers and scientists, to build intelligence into their products. While the barrier to entry for building AI products has been lowered, creating those effective beyond a demo remains a deceptively difficult endeavor." This democratization reminds me of the printing press's impact on biblical texts, which expanded access but also introduced new interpretive challenges. As we navigate these advancements, it's crucial to balance accessibility with responsible use. At Ushur, we've been at the forefront of leveraging LLMs in our backend/knowledge work automation as well as carefully introducing them on the conversational side via our Chatbots and other modes. More will be coming out on that. Read more here: O'Reilly Article #AI #LLM #Technology #Innovation #History #Democratization #ResponsibleAI #Ushur

12

1 Comment