Salman Mohammed

From my conversations, I'm noticing different trends between how AI engineers deploy evaluation models (model as a judge) in production monitoring vs application testing. 💡 Value - Monitoring: provides real time alerts of performance or used to preemptively block problematic outputs from surfacing to the user -Testing: regression testing applications to establish baseline performance before pushing to deployment ⚡ Speed - Monitoring: near real time, requires extremely fast token processing - Testing: batch, async, often processing large test sets, lower token processing speed required 🎯 Accuracy - Monitoring: overindex on false positives, better to flag something as problematic and be wrong than to let something slip between the cracks - Testing: high correlation with human graders, be as accurate and robust as possible 💰 Cost - Monitoring: prefer cheaper models, otherwise the additional inference cost per generation will add up exponentially - Testing: willing to take on higher costs, calculation is accurate testing leads to better performing outputs 🛑 Risk - Monitoring: if your monitoring judge gets something wrong, there is a high consequence because it directly impacts the end user - Testing: if your testing judge gets something wrong, async review gives some potential to remediate the issue At the moment, it seems using model as a judge for testing is a little more popular due to the high cost constraints of constant monitoring, but that is subject to change as more specialized models come out. As the industry matures, I strongly believe we're gonna see specialization between models for monitoring and models for testing. If you're currently using model as a judge techniques, would love to chat more! Let me know what you think. Twitter Handle: https://lnkd.in/eAsQvtiu

26

Come visit Cerebras booth at #Neurips 2024! Along with all the work on the training front, for the first time we will also be talking about some of the ML research we are doing for LLM inference. If you want to know how we are using synthetic data to make our inference efficient and blazing fast, then check out the poster by Vithu Thangarasa.

27

There are lots of awesome works (research papers, tools, libraries,...) right now around Large and Small Language Models (LLMs and SLMs). In you're looking for ways to : * have more control about LLMs output * better ouputs (avoid hallucination) * faster execution  * lower cost I really recommend to look at {guidance} python library which allow you to write guidance program instead of language text prompt (prompt engineering is not a dev thing right :-) The idea with {guidance} is that under the hood, Large Language Models get back the next token and the next token and so on (4 characters each) based on the input prompt (system, user …) based on intensive maths calculation (token probability distribution) which can eventually be change with the temperature parameter. So, what about techniques to avoid generating tokens (because you already know you want it), constrain tokens with possible values and so on... I really like the analogy mentioned by Harsha Nori taking about {guidance} is like a code compiler that transform high level language (python, .net) to specific hardware optimized instructions. here is Microsoft BUILD session talking about {guidance} https://lnkd.in/evpSmbsa #SLM #LLM #AI #MicrosoftAI

13

1 Comment

Our baby is finally heading out to the world on its own!! 🎊 Multimodal AI on RayBan launched to 100% of US/Canadian users yesterday. If you're already a proud owner of a pair of RayBan smart-glasses, it's time to take them out for a joyride. Go ahead, ask them anything that pops into your head. Too lazy to pull out your phone to find the answer? No problem! Your stylish specs are here to save the day. Don't have a pair yet? Well, there's no time like the present! Swing by your nearest RayBan store and ask for a test drive 😎 Personal anecdote time: Those of you who know me personally are very well aware that this has been the most rewarding product of my career. Countless sleepless nights babysitting your models, fretting over every detail and a team who is the perfect balance between fun/work.😛 And there is still a long road to go where we can take this product!! Here is a paper by our team if anyone is interested in the nitty gritty details: https://lnkd.in/g5vqxMen More details about the feature: go.fb.me/ejkqvs

113

20 Comments

I just published Vol. 58 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Learnable Late-Interaction Models for Efficient and Accurate Information Retrieval, from Google 📚 Gaming the AI: How Crafted Content Can Hijack LLM-Powered Search Results, from ETH Zurich 📚 Dynamic Text Segmentation for Enhanced Retrieval in RAG Systems, from Duarte et al. 📚 Improving Collaborative Filtering with Simplified Sampled Softmax Cross-Entropy, from Visa 📚 Automating Quality Assessment in Retrieval-Augmented Generation with GPT-4, from Visa 📚 Efficient Listwise Passage Reranking via Embedding Compression, from Liu et al. 📚 Empowering Smaller LLMs in Passage Re-Ranking through Pre-Filtering, from Polytechnique Montréal 📚 Efficient Online Hyper-Parameter Optimization for RAG Systems, from Microsoft 📚 Decomposed and Distilled Large Language Models for Efficient and Accurate Semantic Search, from Ant Group 📚 Accelerating LLM Reranking with Single-Token Logits, from Reddy et al. #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

73

1 Comment

I just published Vol. 68 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Retrieval-Augmented Generation for Superior Long-Context Question Answering, from NVIDIA 📚 A Near Real-Time Ranking Platform for Modern E-Commerce, from Zalando 📚 A Method for Automated Task-Specific Dataset Creation and Model Fine-Tuning, from Ziegler et al. 📚 A Lightweight Framework for Combining Domain Experts in Information Retrieval, from Lee et al. 📚 Dynamically Determining Compression Rates for Improved RAG Performance, from Zhang et al. 📚 A Comprehensive Study of Pooling and Attention Strategies for Optimizing LLM-Based Embedding Models, from HKUST 📚 Unbiasing Short-Video Recommenders through Causal Inference, from Dong et al. 📚 Non-Parametric Embedding Fine-Tuning for Improved k-NN Retrieval, from Zeighami et al. 📚 A High-Performance Batch Query Architecture for Industrial-Scale Recommendation Systems, from Zhang et al. 📚 Correcting Stale Embeddings for Computationally Efficient Dense Retrieval, from Google DeepMind #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

37

1 Comment

🤖 Staying on top of ML research Machine learning research is a fire hydrant. The field moves so fast, the course you took 5 years ago is probably outdated. To help myself and others stay updated, I thought I could share a few resources that have helped me stay on top of the latest machine learning research papers and trends. 🔍 Emergent Mind (https://lnkd.in/dJK4GbYe) Trending computer science papers from arXiv by topic curated by AI. Provides helpful GPT-4 generated summaries broken down by each section. I find it to be a great way to quickly identify impactful new research to dig into further. 🧠 Bytez (https://bytez.com) AI agent trained on the corpus of AI papers. Summarizes papers and can answer questions on specific topics. I use Bytez to keep browse keep tabs on papers and use it as my first reading step before diving deeper into the actual paper. 📧 Newsletters - 💡 Data Machina (https://lnkd.in/d-mPpbzB) Weekly roundup summaries and papers in MLOps, datasets, deep learning, and AI research - 💡 Deep (Learning) Focus (https://lnkd.in/dsdM2Uhe) In-depth dive into a single topic at a time. (Also a blessing to your LinkedIn feed if you follow the curator 😁) - 💡 AI Tidbits (https://www.aitidbits.ai/) Weekly roundup from 10 other newsletters broken down by subject (language models, vision, audio, industry, open-source packages, etc.) - 💡 AlphaSignal (https://alphasignal.ai/) Short and sweet summaries of top news, papers, and repos. Sometimes it comes with a nice tip that surprises my team (e.g., cutting GPU memory usage with sequential backward calls when using multiple losses in PyTorch).

26

I just published Vol. 67 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 A Comprehensive Study of Noise Types in RAG, from Wu et al. 📚 Lessons from Adobe Express’s Iterative Enhancement of Multi-Modal Search, from Adobe 📚 Dynamic Hard Negative Mining for Enhanced Text Embeddings, from Tencent 📚 Tackling the Unique Challenges of Location-Based Search, from Airbnb 📚 Jina-ColBERT-v2: Enhancing ColBERT for Multilingual Retrieval, from Jha et al. 📚 Enhancing Music Recommendation through Psychology-Informed Session Embeddings, from Deezer 📚 Analyzing [MASK] Token Behavior in ColBERTv2, from RIT 📚 A Critical Evaluation of Traditional Models vs. LLMs in Sequential Recommendation, from Xu et al. 📚 Collaborative Discourse for Complex Information Discovery, from Jiang et al. 📚 A Large-Scale Synthetic Test Collection for Advancing Passage Retrieval Research, from Rahmani et al. #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

34

1 Comment

I just published Vol. 77 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Bridging Visual and Textual Document Understanding with Multi-modal Retrieval, from Bloomberg 📚 Best Practices for Distilling Large Language Models into BERT for Web Search Ranking, from Tencent 📚 Long Context RAG Performance of Large Language Models, from Databricks 📚 A Pairwise Perspective on Softmax Loss for Robust Recommendation, from Zhejiang University 📚 Efficient HTML Processing for RAG Systems, from Tan et al. 📚 A Universal Framework for Multimodal Retrieval with Large Language Models, from NVIDIA 📚 Rationale-Guided Retrieval Augmented Generation for Medical Question Answering, from Korea University 📚 A Reasoning-Based Framework for Document Reranking with Large Language Models, from Salesforce AI Research 📚 Efficient RAG Through Cost-Constrained Chunk Optimization and Dynamic Configuration, from NTU 📚 A Comprehensive Assessment of LLMs in Recommendation, from Jiang et al. #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

31

1 Comment

I can sense the LLM-hype cycle crashing soon. Don’t forget: 1. End-user value matters. Just adding LLM doesn’t make your product better 2. LLMs can’t plan/reason; no amount of monkey patching will overcome this limitation 3. RAG isn’t always better than vanilla BM25 search processed 200+ applications for the AI hackhouse. Many seem to not understand the basic limitations of LLMs + everyone seems to be building similar wrappers on openAI API. I think it’s a function of perverse incentives: it is profitable to hype up your product’s capabilities Ultimately, users don’t care about AI or LLMs. What matters is if you’re solving their problems in a significantly better way v/s all other alternatives. (And remember now all other alternatives are also LLM-powered).

176

15 Comments

If you were handed a few million dollars and a month to train the best LLM, what would you focus on? Check out Yann Dubois's lecture on building LLMs. Rather than diving into model architecture, he emphasizes on the practical aspects of training a "Large" Language Model. Key Takeaways: - Data >> Model Architecture - Scaling Laws for Transformer Models - Use of LLMs to create Supervised Fine-Tuning (SFT) data - Alignment - how to put filters for policy violation - RL from Human Feedback (RLHF): Maximizing response preferences over cloning human behavior   - Evaluation: How do you evaluate models like ChatGPT? (checkout Chatbot Arena LLM Leaderboard - https://lmarena.ai/ Practical advice: Skip over-hyper-tuning — focus on simplicity and scale! #chatgpt #machinelearning #artificialintelligence

216

3 Comments

I've been considering the potential of large language models (LLMs) in the realm of therapy. It's arguable that LLMs can sometimes provide valuable insights that human therapists might miss, given the vast amount of data they are trained on, which allows them to generate thoughtful suggestions. However, their inability to think and reason with unseen data points remains a significant obstacle to fully autonomous AI therapy. What if the first output token is incorrect? The final output can diverge due to the NTP strategy. Could we fine-tune these models with multi-modal inputs—such as speech, video, and text—to better mimic real-world scenarios and assist human therapists?

13

2 Comments

Recently, the TIGER Lab at the University of Waterloo released MMLU-Pro, a more challenging version of MMLU that includes ~10 options per question. It is a more robust and challenging massive multi-task understanding dataset, tailored to rigorously benchmark large language models' capabilities. The dataset contains 12k complex questions across various disciplines. Since the topics are mostly language-agnostic, I translated it into Italian (the translation should be of high quality) to evaluate Italian LLMs. 📂 You can access the dataset here: https://lnkd.in/ed626hgV

35

1 Comment

Very informative post on the main differences between reasoning inquires, versus factual inquires, on LLM capabilities. One of the reasons I enjoy working with GenAI LLMs; is the fascinating contributions and discoveries that are made worldwide, on a daily basis. The opportunities that Generative LLMs can provide for small, medium, and large enterprises are growing everyday. Send me a message on LinkedIn or click the link in my bio if you are interested in contributing to the research and development of the most advanced GenAI models that there is today.

2

I just published Vol. 63 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 A Modular Approach to Retrieval-augmented Generation, from Gao et al. 📚 A Survey on Retrieval Methods in Modern Recommender Systems, from SJTU 📚 A Self-Guided Approach for More Robust RALMs, from Baidu 📚 Efficient Retrieval Planning for Complex Conversational AI Systems, from Amazon 📚 Efficient Long-Context Multilingual Text Retrieval Models, from Alibaba 📚 Integrating Multimodal Representations in E-commerce Recommendations, from Alibaba 📚 A Systematic Review of Recommender System Challenges and Opportunities, from Ma et al. 📚 A Multi-Agent Framework for Advanced Web Information Seeking and Integration, from Chen et al. 📚 Efficient Dimensionality Reduction for LLM Embeddings in Information Retrieval, from Google Cloud AI 📚 Improving Text Embeddings in Resource-Efficient Language Models, from Tsinghua University #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

44

3 Comments

I just got my own copy of “Building LLMs for Production” by Louis-François Bouchard and Louie Peters. As I am getting more into applications of GenAI I expect this book to become one of my go-to resources in the upcoming months. https://amzn.to/3Yv5FmL #AI #GenAI #LLMs

2,250

51 Comments

ArXiv alert! Interesting and recent applied research done by scientists in our Foundation Models team at Netflix Research - Sejoon Oh, Moumita Bhattacharya, and Yesu Feng - is available on ArXiv. Here we motivate the problem of predicting users' session intents and propose a solution for it using Hierarchical Multi-Task Learning. Not only do we get good predictions of the (proxy) intents but also show that the architecture improves the foundation model. https://lnkd.in/gEGQRhg9 Hope you all like it. #foundationmodels #recsys #machinelearning #research

157

1 Comment

I just published Vol. 48 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 A Systematic Survey of Retrieval-Augmented Generation Techniques, from York University 📚 Navigating the Evaluation Landscape of Generative Information Retrieval, from Alaofi et al. 📚 Leveraging RAG to Enhance the Trustworthiness of Generative AI for Enterprise Workflow Automation, from ServiceNow 📚 Exploring the Interplay Between Large Language Model Priors and Externally Sourced Content in Retrieval-Augmented Systems, from Stanford University 📚 A Practical Roadmap for Efficient Fine-Tuning on Proprietary Enterprise Data, from HCLTech 📚 Leveraging Look-Alike Users and Recalled Items to Improve Click-Through Rate Prediction, from Huawei 📚 Leveraging Collaborative Knowledge in Large Language Models for Robust and Versatile Recommendation, from KAIST 📚 A Study on How LLM-Generated Content Infiltrates and Reshapes Retrieval-Augmented Generation Systems, from UCAS 📚 Leveraging Rotary Position Embeddings for Extended Context Windows in Text Embedding Models, from Microsoft 📚 Taxonomy and Analysis of Sensitive User Queries in Generative AI Search, from Naver #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

71

3 Comments