Alex Wang

Even when accurate context is retrieved, can LLMs extract the preferred answer when layers of conditions or constraints are applied across conversation turns? 🤔 🌟 Thrilled to share my recent work: RAD-Bench: Evaluating Large Language Models' Capabilities in Retrieval Augmented Dialogues, developed during my internship at MediaTek Research 聯發創新基地 ! With RAG, SAG, and tool-using becoming prevalent, it's crucial to evaluate how models handle progressively accumulated constraints and external information in context-rich scenarios. Existing benchmarks either assess LLMs' chat abilities in multi-turn dialogues or their use of retrieval for augmented responses in single-turn settings. To bridge the gap, we proposed RAD-Bench, which to the best of our knowledge is the first benchmark to assess LLMs' ability to follow user instructions in multi-turn dialogues while effectively utilizing retrieved context. Our benchmark focuses on two key abilities: Retrieval Synthesis and Retrieval Reasoning. We constructed a pipeline using LLMs to generate, select, and synthesize synthetic questions and retrieved contexts, resulting in 89 high-quality multi-turn samples (267 total turns). We evaluated popular LLMs like GPT-4, Llama, Gemma, Mistral, Deepseek, and Breeze using LLM-as-a-Judge with tailored prompts. Our findings show that as conditions accumulate, models struggle more to identify key information from context. Comparisons with Chatbot Arena reveal that RAD-Bench effectively distinguishes LLM performance in context-rich, augmented dialogues, demonstrating that models with similar performance in regular multi-turn conversations may differ in retrieval-augmented scenarios. For detailed insights, check our paper on arXiv: https://lnkd.in/gKT8ACze Heartfelt thanks to my mentor Feng-Ting Liao for guidance and discussions, Mu-Wei Hsieh for additional experiments and result visualization, and Mark for paper revisions. Your contributions were instrumental in bringing this work to fruition! #LLM #MultiTurn #RAG #Benchmark #Research #Internship

59

4 Comments

💡 Making LLMs More Accessible: A Game-Changer for Students and Researchers Today, I was wondering: if there are so many challenges in fine-tuning LLMs, how can someone without access to physical or server-based GPUs implement such techniques? This is especially relevant for students and researchers with limited resources. Then, I came across a fascinating research paper introducing BitNet b1.58, which could provide a practical solution to this dilemma. The concept is groundbreaking: transforming LLMs by representing weights in ternary values {-1, 0, 1}, enabling significant reductions in memory and computational requirements while retaining performance comparable to full-precision models (e.g., FP16). This innovation isn't just about efficiency—it's about redefining accessibility in AI. For example, BitNet b1.58 achieves: Memory Savings: Up to 3.55x less memory usage compared to LLaMA models. Speed Improvements: 2.71x faster inference at 3B model sizes. Energy Efficiency: 71.4x lower energy consumption for matrix multiplication. Comparison Table Model: LLaMA LLM Size: 700M Memory (GB): 2.08 (1.00x) Latency (ms): 1.18 (1.00x) Perplexity (PPL): 12.33 Model: BitNetb1.58 Size: 700M Memory (GB): 0.80 (2.60x) Latency (ms): 0.96 (1.23x) Perplexity (PPL): 12.87 A Key Thought: If implementations like BitNet b1.58 become widely available, they could revolutionize how LLMs are deployed, especially in resource-constrained environments like edge devices and mobile platforms. Imagine bringing the power of LLMs to more devices and applications than ever before, empowering students and researchers to innovate without the need for high-end infrastructure. To know more, please go read the paper!!! #LLM #AI #BitNet #Innovation #Quantization #Education #theiotacademy #DeepLearning #NLP #machinelearning

6

📣 📣 📣 [CFP; MRR 2024 Workshop @ SIGIR 2024 Conference]We're excited to invite researchers and practitioners across the spectrum of multimodal modeling, representation learning, and retrieval  to participate in our workshop on "Multimodal Representation and Retrieval." This event is proudly organized in association with SIGIR 2024. Multimodal data is available in many applications like e-commerce production listings, social media posts and short videos. However, existing algorithms dealing with those types of data still focus on uni-modal representation learning by vision-language alignment and cross-modal retrieval. In this workshop, we target to bring a new retrieval problem where both queries and documents are multimodal. With the popularity of vision language modeling,  large language models (LLMs), retrieval augmented generation (RAG), and multimodal LLM, we see a lot of new opportunities for multimodal representation and retrieval tasks. This event will be a comprehensive half-day workshop focusing on the subject of multimodal representation and retrieval. The agenda includes keynote speeches, oral presentations, and an interactive panel discussion. Our objective with this workshop is to capture the interest of researchers in this emerging challenge. By highlighting the novelty and significance of the problem, we aim to attract researchers who are eager to explore and contribute to this field. Join us as we welcome esteemed speakers including Tat-Seng Chua (National University of Singapore), and Dinesh Manocha (University of Maryland), who will share their invaluable insights and expertise. We invite original research & industrial application papers that present research on multimodal data representation and retrieval. 🔗 Workshop website: https://lnkd.in/gcpZtrdc 📅 Submission Deadline: May 5, 2024 (11:59 pm, AOE) 📅 Workshop Date: July 18th, 2024 🌆 Workshop Location: Washington D.C., USA Join us to shape the future of multimodal representation, retrieval, and related applications! ☀ My co-organizers Arnab Dhua (Amazon), Douglas Gray (Amazon), I. Zeki Yalniz (Meta), Tan Yu (Nvidia), Mohamed Elhoseiny (KAUST), Bryan A. Plummer (Boston University) and I look forward to hosting you.

74

4 Comments

I just watched the "InternLM - A Strong Agentic Model?" video of Sam Witteveen [1]. I really enjoyed his comprehensive overview of InternLM2.5, a 7B parameter model (Shanghai AI Laboratory), which was released four days ago. It is available on Hugging Face [2] and Ollama [3]. This model is now ranked one among all models below 10B/12B parameters. After reviewing their GitHub repository, showing us how this model has outstanding reasoning capabilities and a 1M context window [4], Sam Witteveen points out the LMDeploy capability and says that it has stronger tool use, function calling, reflection, and more agentic usages [5]. Introducing Lagent [6], Sam Witteveen emphasizes examples shown in their GitHub repository and says: "This is like an agentic framework that may use this particular model." Afterward, he talks about the corresponding paper as of then (InternLM2), resulting in how big of a dataset is used (10 million instruction data instances) and how Conversation has a little more than 50% of data topics, and Math and NLP Tasks are coming in second and third place respectively by 18% and 16% in Supervised Fine-Tuning (SFT). In a comparison between the LLaMA2 and InternLM2, the paper illustrates the difference and shows "how the Conditional Reward Model incorporates different system prompts for different kinds of preferences." [7] With a quick recap of the InternLM2.5 model being available on both HuggingFace and Ollama [2, 3], he started coding with HuggingFace and Ollama implementation on his Colab [8]. I find his guide and thorough explanation very useful and encourage you to check out his YouTube video for better understanding [1]. P.S. It is currently 1st among models below 12B params, ranked 2nd among models below 13B params, and ranked 3rd among models below 34B params according to HuggingFace Open-LLM Leaderboard [9]. [1]: https://lnkd.in/dGrSnMDt [2]: https://lnkd.in/d9z4C7gu [3]: https://lnkd.in/ddXCRVxM [4]: https://lnkd.in/dMy7GxFy [5]: https://lnkd.in/dDd3EeTd [6]: https://lnkd.in/dPUBTfbB [7]: https://lnkd.in/da3yQV-G [8]: https://lnkd.in/dbu4Cn7K [9]: https://lnkd.in/dkuepwgW

14

How can LLMs aid in the process of building Simulation models? Some weeks ago, my (first) Journal article was published! There, we examined the potential of prompting ChatGPT, to generate functional Simulation model code from a prose-based narrative. Please check it out! #LLMs #AI #GenAI

8

1 Comment

🌍 Thrilled to have virtually presented our paper, "Information Parity: Measuring and Predicting the Multilingual Capabilities of Language Models," at EMNLP 2024 last week! This work builds upon and extends our earlier research on Compression Parity (CP), which was presented at the ICML 2024 Theoretical Foundations of Foundation Models workshop. While the metric itself remains fundamentally the same—now renamed as Information Parity (IP)—this new paper introduces significant advancements in analysis and evaluation. What’s new in this iteration? We conduct extensive statistical analysis to validate IP and demonstrate its superiority over alternative metrics, such as tokenization-based measures (e.g., Tokenization Parity and Tokenizer Fertility) and the proportion of languages in training data. Notably, we show that: • IP consistently outperforms these metrics in predicting multilingual model benchmarks performance, such as multilingual ARC and MMLU. • IP overcomes predictive biases inherent to tokenization, such as those associated with different script families, providing reliable multilingual performance predictions. • Tokenization Parity and Tokenizer Fertility are captured within the explained variance of IP, further highlighting its ability to holistically represent multilingual performance. In essence, IP offers a robust, scalable, and task/prompt-agnostic way to evaluate LLMs, even for low-resource languages where existing methods often fall short. A huge thank you to my co-author, Alon Kipnis, for his invaluable contributions and mentorship throughout this work! Read the full paper here: https://lnkd.in/dzreCQ3N

23

1 Comment

Our newest paper on Recommender System🎉 I am excited to share with you our newest paper titled “Deep recommendation with iteration directional adversarial training”https://lnkd.in/g_mnU5ay

2

I am excited to share that our intern Haiyi Mao's FCR paper will appear in #NeurIPS 2024. Recognizing that cell responses vary based on biological context (like genetic background or cell type), the FCR method is designed to capture complex interactions between drugs and cellular contexts. Built on identifiable deep generative models and recent advances in non-linear ICA theory, FCR disentangles cellular responses into covariate-specific, treatment-specific, and interaction-specific components. The team will attend #NeurIPS please come to talk to us if you are interested in AI in Drug Discovery at Genentech AI. Paper link: https://lnkd.in/gAkXHZNn GitHub link: https://lnkd.in/g6s2dSWZ

58

2 Comments

OpenCoder: An Open-Source Code LLM Redefining Model Efficiency 🌟 The OpenCoder team has launched a game-changer in the world of code language models. Unlike many LLMs that focus on sheer size, OpenCoder brings something unique to the table: a data-focused training approach. By putting quality before volume, OpenCoder’s 1.5B and 8B models rival some of the biggest players, like DeepSeekCoder and QwenCoder, with only a fraction of their token count. The core strength of OpenCoder lies in its meticulous data processing pipeline and two-stage fine-tuning. With a training dataset composed of 2.5 trillion tokens, 90% raw code and 10% code-related web data, OpenCoder refines every aspect of its corpus. Advanced filtering and deduplication strategies, applied to 607 programming languages, optimize model efficiency and precision, ensuring high-quality data drives its training. Key Features: - Complete Training Transparency: Open-source datasets and training pipelines enable reproducibility and further research. - Instruction Fine-Tuning: Two-stage tuning sharpens both core CS concepts and real-world coding skills, prepping OpenCoder for production-level code generation. - Available on Ollama: Accessibility in deployment means more avenues for experimentation and improvement. OpenCoder's data-driven strategy showcases the potential of a leaner approach. By focusing on quality data rather than quantity alone, OpenCoder achieves file-level deduplication that outperforms traditional repo-level methods, making it both efficient and robust. 🚀 Read more here: https://lnkd.in/eEnG3yJA #OpenSource #MachineLearning #CodeLLM #AI #ModelEfficiency #DataQuality #Innovation #CodeGeneration #OpenCoder #TechNews #AIResearch #Programming #LargeLanguageModels #ArtificialIntelligence

1

Excited to share our recent work on applying context parallelism for scalable million-token inference, detailed in the paper: https://lnkd.in/gCPj5eaK . In this work, we present context parallelism for long-context large language model inference, which achieves near-linear scaling for long-context prefill latency with up to 128 H100 GPUs across 16 nodes. Particularly, our method achieves 1M context prefill with Llama3 405B model in 77s (93% parallelization efficiency, 63% FLOPS utilization) and 128K context prefill in 3.8s. We have developed two lossless exact ring attention variants: pass-KV and pass-Q, which cater to a wide range of use cases with state-of-the-art performance, including full prefill, persistent KV prefill, and decode. Our benchmarks on H100 GPU hosts interconnected with RDMA and TCP demonstrate similar scalability for long-context prefill, indicating that our method scales effectively in common commercial data centers with medium-to-low inter-host bandwidth.

223

6 Comments

Just posted my working paper "Beyond Collaborative Filtering: A Relook at Task Formulation in Recommender Systems" on ArXiv https://lnkd.in/gbaZwrsH I began reconsidering RecSys when I sensed that the baseline popularity method wasn't suitable ( https://lnkd.in/gtYHsuZ2 ). Simply counting interactions in the training set failed to capture the temporal trends, the common understanding of popularity. This led me to realize that many RecSys models and offline evaluations lacked a time dimension, raising concerns about data leakage ( https://lnkd.in/g7fpuyPH ). I then questioned the suitability of a widely used benchmark dataset ( https://lnkd.in/gQhBWhzS ). This opinion paper ( https://lnkd.in/gbaZwrsH ) might provide some answers to why academic researchers continue to use these datasets and formulate the RecSys problem in the current way. The whole journey led to a collection of "negative results". ( https://lnkd.in/g_Hg7KKt )

47

🔍 Ever wondered if there's a unified framework for Retrieval-Augmented Generation (RAG)? We've formalized the retrieval enhancement paradigm with consistent notations, synthesizing research across all machine learning domains. Check out our new preprint: https://lnkd.in/e5iJv99z 📍Retrieval augmentation isn't just for language models! It can be generalized across machine learning, extending its impact far beyond language generation to areas like speech processing, computer vision, time series prediction, and computational biology. 🌟 We introduce Retrieval-Enhanced Machine Learning (REML), a formal framework that integrates retrieval mechanisms into machine learning models to enhance their performance. Our key contributions: 1️⃣ Synthesis and Unification We've synthesized existing research and unified it into a coherent framework, providing consistent notations that bridge various ML domains. 2️⃣ Modularization of REML Retrieval-Enhanced Machine Learning (REML) is modularized into distinct components: Querying, Searching, Presentation, Consumption, Storing, Optimization, and Evaluation. This modular approach makes it easier to understand and implement. 3️⃣ Enriching with IR Research Information Retrieval (IR) research is often overlooked by NLP researchers. Each modularized part of REML is enriched by the rich history and foundational principles of IR research. 4️⃣ Proposing Future Research By synthesizing and unifying existing work, we identified numerous gaps and areas for improvement. We propose numerous future research opportunities in this emerging field of REML. Incredible work with Alireza Salemi, Andrew Drozdov, Fernando Diaz, and Hamed Zamani

69

Exciting News! ✨ I’m thrilled to share that my latest paper, 'Bayesian Optimized Machine Learning Model for Automated Eye Disease Classification from Fundus Images' has been officially published today in Computation! This paper dives into four types of eye diseases and introduces an advanced machine learning model, optimized with Bayesian techniques, to classify these diseases from fundus images with enhanced accuracy. After extensive experimentation, it’s been an incredible journey from research to publication. A huge thank you to me for my invaluable support and guidance throughout this process. Just kidding! 😆 A heartfelt thank you to everyone who contributed to this paper and supported me throughout this process! If you’re interested in relevant field or topic, you can check out the paper here: https://lnkd.in/gwnFda4N 🔮 #Research #MachineLearning #AI #EyeDisease #BayesianOptimization #HealthcareInnovation #Ophthalmology #MedicalAI

12

3 Comments

Take a look at my article! https://lnkd.in/daWe_GRr This article is very interesting for application in area of Engineering fuzzy nature problem and also in medical science to model bacterial/ virous effects on human body. Author is my research scholar T. Jamir and myself

8

In Meta FAIR CodeGen team we released 2 papers for multi-turn code generation in October: 1. RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning https://lnkd.in/eDAahSmu. It's a training time effort. It's again proven that simple design works the best. We achieved impressive results with Llama 3.1 70B using PPO and nothing-else fancy (but it does need quite an amount of engineering efforts to make it scale). 2. What Makes Large Language Models Reason in (Multi-Turn) Code Generation? https://lnkd.in/emdFBAKP. It's an inference time effort (though we do minimal fine-tuning to make the model have CoT-level ability without CoT prompting). We group all the CoTs and test > 60 combinations of it across multiple model sizes and series up to 405B. So many interesting insights there (e.g. bad CoT hurts the performance, fine-grained debugger information skews the diversity). You might already spot what we are cooking right now :)

67

1 Comment