Eric Nguyen

IndicGenBench Google Research India recently released IndicGenBench, a multilingual benchmark to evaluate generation capabilities of LLMs on 29 Indic languages spanning 13 writing scripts and 4 language families. Extended the datasets in Cross-lingual Summarization, Machine Translation, Multi-lingual Question Answering, and Cross-lingual Question Answering, the team has collected human translations for English examples into target Indic languages, thereby extending the scope and applicability of evaluation metrics in this domain. One of the key insights from their study is the analysis of token fertility across all Indic languages within IndicGenBench. Token fertility, representing the average number of sub-words that a word is broken down into by the tokenizer, varies significantly across languages Some languages have simple breakdowns, while others are more complex. Now, why does this matter? Well, it affects how well the language models work. Languages with more complex breakdowns might struggle because they can't use as many examples to learn from. They found that languages with simpler breakdowns can use more examples effectively compared to those with complex breakdowns. #LLMs #GenAI #IndicGenBench #AI #IndicLanguages #IndicDatasets #multilingual

15

1 Comment

In LLMs, long input sequences presents a significant challenge in detecting hallucinations using smaller, cheaper models. One of the key innovations (that blows past LLM-as-judge techniques) introduces a novel windowing mechanism specifically designed to address this issue. Consider this -- conventionally, the standard procedure for models with limited context lengths includes Segmentation, Localized Prediction and Aggregation. This traditional approach proves suboptimal for hallucination detection. e.g. a scenario where the language model generates a statement, "X", which is substantiated by context, "Y". If segments are created such that "X" and "Y" reside in separate chunks, the model remains oblivious to the mutual reinforcement between "X" and "Y", thereby impairing accurate hallucination detection. The approach we take circumvents the limitation by deploying a more refined windowing strategy (RAG context and the model-generated responses getting divided into small overlapping windows). This allows a smaller, auxiliary model to evaluate every possible pair of possible context and response windows. This shift in the windowing methodology offers a more robust framework for hallucination detection, advancing the frontier of reliability in RAG deployments. If you're interested in learning more, read our research here: https://lnkd.in/gFAqr7wB #llmhallucinations #llms #generativeai #llmevaluation #llmevals #llmbenchmarking

38

2 Comments

OpenCoder: An Open-Source Code LLM Redefining Model Efficiency 🌟 The OpenCoder team has launched a game-changer in the world of code language models. Unlike many LLMs that focus on sheer size, OpenCoder brings something unique to the table: a data-focused training approach. By putting quality before volume, OpenCoder’s 1.5B and 8B models rival some of the biggest players, like DeepSeekCoder and QwenCoder, with only a fraction of their token count. The core strength of OpenCoder lies in its meticulous data processing pipeline and two-stage fine-tuning. With a training dataset composed of 2.5 trillion tokens, 90% raw code and 10% code-related web data, OpenCoder refines every aspect of its corpus. Advanced filtering and deduplication strategies, applied to 607 programming languages, optimize model efficiency and precision, ensuring high-quality data drives its training. Key Features: - Complete Training Transparency: Open-source datasets and training pipelines enable reproducibility and further research. - Instruction Fine-Tuning: Two-stage tuning sharpens both core CS concepts and real-world coding skills, prepping OpenCoder for production-level code generation. - Available on Ollama: Accessibility in deployment means more avenues for experimentation and improvement. OpenCoder's data-driven strategy showcases the potential of a leaner approach. By focusing on quality data rather than quantity alone, OpenCoder achieves file-level deduplication that outperforms traditional repo-level methods, making it both efficient and robust. 🚀 Read more here: https://lnkd.in/eEnG3yJA #OpenSource #MachineLearning #CodeLLM #AI #ModelEfficiency #DataQuality #Innovation #CodeGeneration #OpenCoder #TechNews #AIResearch #Programming #LargeLanguageModels #ArtificialIntelligence

1

I enjoyed this blog post on agents. Some historical context is always great. It argues that we have much better reasoning engines with LLMs, and by building engineering around them, we can also make agentic workflows more reliable. https://lnkd.in/dBFBS9zM

38

2 Comments

Heterogeneous graph data is ubiquitous; different edge relationships and attribute types come up in social/epidemic graphs, molecular/protein graphs, and buyer behavior graphs. This recent paper addresses heterogeny in graph neural network prediction through a topological tool called sheafs, which aggregate properties of edges and vertices according to a pre-defined (or learned) function. Definitely have a look at this recent arXiv paper out of Cambridge if you are working with these types of graph neural networks! https://lnkd.in/eZHHhtuN

47

1 Comment

We just released a blog on whether the use of multimodal feature transfer can detect deception (phenomena like sarcasm, irony, and condescension) in conversation. The short answer... Yes. This particular write-up profiles a subset of our work at Symbl.ai on multimodal large language models, and is research led by one of our summer interns, Benjamin R. (from Prof. Larry Heck's AVA Lab at Georgia Institute of Technology). We have a paper currently under submission on this work, and we can't wait to share the full details with all of you soon. Blog Link: https://lnkd.in/g6m_wFDi #multimodal #featuretransfer #LLMs #deception #conversation #sarcasm #irony #language #AI #ML #NLP #LargeLanguageModels #SummerInternship #ResearchPapers #ACL #Llama2 #SymblDotAI

16

It is well understood that when generating text, Large Language Models (LLMs) predict multiple possible next words, each associated with a probability. An effective decoding strategy is essential for selecting the next token from the probability distribution, ensuring coherent text generation. For an insightful discussion on the various decoding strategies employed in LLMs, explore the video "LLMs: Hands-on Decoding Strategies", which delves into key techniques like greedy search, beam search, top-k sampling, and nucleus (Top-p) sampling. LLMs: Hands-on Decoding Strategies --- https://lnkd.in/gVk-5Rim #LLMs #AI #MachineLearning #NLP #DecodingStrategies #ArtificialIntelligence #LanguageModels #TechTutorial #AICommunity #Innovation #DeepLearning

29

Large time-series models (LTMs) enjoy similar power-law scaling behaviour to LLMs. We just put out a paper (https://lnkd.in/givY528D) establishing power-law like scaling-laws for large time-series models as a function of data, compute, and model size. Similar scaling-laws for LLMs (from the landmark Kaplan et al. paper https://lnkd.in/g9KHYN9u) have provided key guidance in allocating enormous resources for predictable - and eventually breakthrough - performance gains. The demonstration of similarly favourable scaling behaviour for large time-series models provides both a motivation and guide in the pursuit of foundation models for time-series forecasting. Foundation models for time-series are coming (with enough data and compute). Thanks to Thomas Edwards, James Alvey, Benjamin Wandelt and Nam Nguyen for the hard work!

48

1 Comment

Let's have a look at the Best Paper Honorable Mention of ACM SIGGRAPH 2024! High quality asset generation "CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets" is a great paper published by DeemosTech, Huazhong University of Science and Technology and ShanghaiTech University. They introduce CLAY, a 3D geometry and material generator designed to effortlessly transform human imagination into intricate 3D digital structures. CLAY supports classic text or image inputs as well as 3D-aware controls from diverse primitives (multi-view images, voxels, bounding boxes, point clouds, implicit representations, etc). At its core is a large-scale generative model composed of a multi-resolution Variational Autoencoder (VAE) and a minimalistic latent Diffusion Transformer (DiT), to extract rich 3D priors directly from a diverse range of 3D geometries. Specifically, it adopts neural fields to represent continuous and complete surfaces and uses a geometry generative module with pure transformer blocks in latent space. They present a progressive training scheme to train CLAY on an ultra large 3D model dataset obtained through a carefully designed processing pipeline, resulting in a 3D native geometry generator with 1.5 billion parameters. For appearance generation, CLAY sets out to produce physically-based rendering (PBR) textures by employing a multi-view material diffusion model that can generate 2K resolution textures with diffuse, roughness, and metallic modalities. They demonstrate using CLAY for a range of controllable 3D asset creations, from sketchy conceptual designs to production ready assets with intricate details. Even first time users can easily use CLAY to bring their vivid 3D imaginations to life, unleashing unlimited creativity. You can test it with Rodin-Gen-1 already! https://lnkd.in/entaZjtR All links in comments below #generativeai #mesh #cad #assets #vfx #assetsgeneration #diffusionmodels #neuralradiance #machinelearning #deeplearning #futurism #computervision #graphics #graphicsengine

26

4 Comments

My lecture "Aligning Open Language Models" for Stanford CS25, Transformers United, is now online! I'm very excited about this. We review all the major models, datasets, papers, and events that have impacted my job and outlook in the last 18 months or so since ChatGPT. I don't normally shill my stuff too hard, but this was one of my favorite lectures I've given since working on RLHF. It's worth watching for so many different reasons. The lecture abstract: Since the emergence of ChatGPT there has been an explosion of methods and models attempting to make open language models easier to use. This talk retells the major chapters in the evolution of open chat, instruct, and aligned models, covering the most important techniques, datasets, and models. Alpaca, QLoRA, DPO, PPO, and everything in between will be covered. The talk will conclude with predictions and expectations for the future of aligning open language models. Watch it here: https://lnkd.in/gUMvjMw2 Slides: https://lnkd.in/gdAYNfdh

578

9 Comments

Brand new paper on September 5th, 2024 on Mamba and Physics-informed machine learning. State-space models are accurate and efficient neural operators for dynamical systems Zheyuan Hu, Nazanin Ahmadi Daryakenari, Qianli Shen, Kenji Kawaguchi, George Karniadakis arXiv: https://lnkd.in/gnVbq5rm Physics-informed machine learning (PIML) has emerged as a promising alternative to classical methods for predicting dynamical systems, offering faster and more generalizable solutions. However, existing models, including recurrent neural networks (RNNs), transformers, and neural operators, face challenges such as long-time integration, long-range dependencies, chaotic dynamics, and extrapolation, to name a few. To this end, this paper introduces state-space models implemented in Mamba for accurate and efficient dynamical system operator learning. Mamba addresses the limitations of existing architectures by dynamically capturing long-range dependencies and enhancing computational efficiency through reparameterization techniques. To extensively test Mamba and compare against another 11 baselines, we introduce several strict extrapolation testbeds that go beyond the standard interpolation benchmarks. We demonstrate Mamba's superior performance in both interpolation and challenging extrapolation tasks. Mamba consistently ranks among the top models while maintaining the lowest computational cost and exceptional extrapolation capabilities. Moreover, we demonstrate the good performance of Mamba for a real-world application in quantitative systems pharmacology for assessing the efficacy of drugs in tumor growth under limited data scenarios. Taken together, our findings highlight Mamba's potential as a powerful tool for advancing scientific machine learning in dynamical systems modeling.

98

How do language models comprehend gibberish inputs? Our recent work with James Zou focuses on understanding the mechanisms by which LLMs can be manipulated into responding with coherent target text to seemingly gibberish inputs. Paper: https://lnkd.in/gA9Mjqc4 A few takeaways: In this work we show the prevalence of nonsensical prompts that induce LLMs to generate specific and coherent responses, which we call LM Babel. We examine the structure of Babel prompts and find that despite their high perplexity, these prompts often contain nontrivial trigger tokens, maintain lower entropy compared to random token strings, and cluster together in the model representation space. We find that the efficiency of these prompts largely depends on the prompt length as well as target text’s length and perplexity. We show that reproducing harmful texts with aligned models is not only feasible but, in some cases, even easier compared to benign texts, while fine-tuning language models to forget specific information complicates directing them towards unlearned content.

31

When selecting the answer for an multi choice question, LLMs prefer the choices in a given position, or with a specific choice symbol. This is known as selection bias. It hugely reduce the capability of using LLM to do labeling. Luckily, selection bias has been tackled by our intern Froilan (Hyeong Kyu) during his summer internship! Check this amazing work lead by him! Here are some key findings: 1. Selection bias is more extreme when the model’s response is incorrect. 2. Most of selection bias observed to be in the final output layer of the decode. Thus, pruning those nodes reduce selection bias. 3. Adding an "I don't know" option help LLM reduce selection bias even for black box LLM! Amazon Science

24

How can LLMs aid in the process of building Simulation models? Some weeks ago, my (first) Journal article was published! There, we examined the potential of prompting ChatGPT, to generate functional Simulation model code from a prose-based narrative. Please check it out! #LLMs #AI #GenAI

8

1 Comment

Implementing Reinforcement Learning (RL) in the context of Large Language Models (LLMs) posed challenges, particularly in designing reward engineering and penalization components. In traditional RL, reward engineering is straightforward, often defined through a scalar reward function. However, implementing the same concept in RL with LLMs required a critic model that evaluates the questions and answers, critiquing them in a manner that simulates the reward function. A recent research paper on Natural Language Reinforcement Learning (NLRL) provides an intriguing perspective on the reward problem in RL by translating traditional RL concepts into natural language constructs. Here's a summary of how it addresses the reward problem: Language-Based Value Functions: Unlike traditional RL, which uses scalar rewards, NLRL employs language-based value functions. These functions articulate the effectiveness of policies and actions in a more interpretable, natural language format. This approach addresses the need for richer, more informative feedback than simple numerical scores. Iterative Refinement of Language Policies and Critics: NLRL incorporates a structured training pipeline where language-based policies and critics are iteratively refined using textual feedback. This method allows the system to handle the complexities of defining rewards in natural language, improving its understanding and evaluation of actions over time through textual feedback loops. Use of Large Language Models (LLMs): The framework leverages the advanced capabilities of LLMs to interpret, generate, and evaluate complex language-based information. This is essential for implementing language-based value functions and policies, enabling the system to generate and refine responses that align closely with desired outcomes. Distilling Enhanced Actions and Evaluations: NLRL employs a process where enhanced evaluations from the language-based value function approximator and improved actions from the language policy improvement operator are distilled. This allows for continuous refinement and learning. Compatibility with Existing LLMs: The framework is designed to be compatible with current LLMs, enabling it to leverage pre-existing language models to process and generate linguistic data effectively. This compatibility is crucial for translating the traditional reward engineering process into a language-based framework. The NLRL framework's introduction of natural language as a medium for representing and evaluating policies and rewards offers a novel solution to the reward problem in RL. By shifting from numerical reward systems to a more flexible, expressive, and intuitive system based on language, NLRL enhances interpretability and adaptability, paving the way for more diverse RL applications. Innovation Hacks AI Inc. #llm #refincorcementlearning #ai

4

🚀 On Reasoning in LLMs Paper: Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization" by Wang et al. https://lnkd.in/gZ6GB2-N It offers profound insights into how transformer models learn to reason implicitly. Here are the key takeaways: 1️⃣ Grokking in Reasoning Tasks: The study focuses on two reasoning tasks: composition and comparison. Transformers can learn these tasks, but only through "grokking" - a phenomenon where generalization occurs after extended training beyond overfitting. 2️⃣ Mechanistic Analysis: Using causal tracing and logit lens techniques, the researchers peered into the model's internals during grokking. They identified specific "circuits" that form: - A sequential circuit for composition - A parallel circuit for comparison This difference in circuit structure explains varying levels of systematic generalization across tasks. 3️⃣ Efficiency Hypothesis: Grokking is explained as a transition from a memorizing circuit to a more efficient generalizing circuit. This transition is driven by: - The model's implicit bias towards efficiency - Explicit regularization (e.g., weight decay) 4️⃣ Parametric vs. Non-parametric Memory: In complex reasoning tasks with large search spaces, the grokked transformer (parametric memory) outperformed state-of-the-art language models like GPT-4-Turbo and Gemini-1.5-Pro (non-parametric memory). 5️⃣ Implications for Architecture Design: The findings suggest potential improvements to transformer architectures, such as incorporating cross-layer memory sharing mechanisms to enhance generalization capabilities. #grokking #transformers #llms #gpt4 #claude #cmu

10

Proposes Minstrel, a multi-generative agent system with reflection capabilities to automate structural prompt generation. It also presents LangGPT, an extensible framework for designing prompts. Minstrel is built on top of LangGPT. "Experiments demonstrate that structural prompts (either generated by Minstrel or written manually) perform better in guiding LLMs to perform tasks." Paper: https://lnkd.in/eWFNUkkd

27

1 Comment

Is GPT-4o good at table recognition? Not quite! OCR looks fine, but tripped by spanning cell table structure! 🤩 However, our UniTable, with only 124M param trained by self-supervision, shows great understanding! 👉 UniTable paper: arxiv.org/abs/2403.04822 👉 UniTable code: https://lnkd.in/e4a7YMxJ #MachineLearning #GPT-4o #Table #Document

85

8 Comments