Rishabh Jain

IndicGenBench Google Research India recently released IndicGenBench, a multilingual benchmark to evaluate generation capabilities of LLMs on 29 Indic languages spanning 13 writing scripts and 4 language families. Extended the datasets in Cross-lingual Summarization, Machine Translation, Multi-lingual Question Answering, and Cross-lingual Question Answering, the team has collected human translations for English examples into target Indic languages, thereby extending the scope and applicability of evaluation metrics in this domain. One of the key insights from their study is the analysis of token fertility across all Indic languages within IndicGenBench. Token fertility, representing the average number of sub-words that a word is broken down into by the tokenizer, varies significantly across languages Some languages have simple breakdowns, while others are more complex. Now, why does this matter? Well, it affects how well the language models work. Languages with more complex breakdowns might struggle because they can't use as many examples to learn from. They found that languages with simpler breakdowns can use more examples effectively compared to those with complex breakdowns. #LLMs #GenAI #IndicGenBench #AI #IndicLanguages #IndicDatasets #multilingual

15

1 Comment

It is well understood that when generating text, Large Language Models (LLMs) predict multiple possible next words, each associated with a probability. An effective decoding strategy is essential for selecting the next token from the probability distribution, ensuring coherent text generation. For an insightful discussion on the various decoding strategies employed in LLMs, explore the video "LLMs: Hands-on Decoding Strategies", which delves into key techniques like greedy search, beam search, top-k sampling, and nucleus (Top-p) sampling. LLMs: Hands-on Decoding Strategies --- https://lnkd.in/gVk-5Rim #LLMs #AI #MachineLearning #NLP #DecodingStrategies #ArtificialIntelligence #LanguageModels #TechTutorial #AICommunity #Innovation #DeepLearning

29

EMNLP 2024 results are out, we got in three papers, congrats to students :) First paper of many, Vatsal, Pranshu, Suyash, Janit, Kunal, Srija and Adnan, Congrats 🎉 🎊 🍾 🎈 Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets (main) - Vatsal Gupta, PRANSHU PANDYA, Tushar Kataria, Vivek Gupta, Dan Roth tldr: make models robust on multiple adversarial challenge set simultaneously  arxiv: https://lnkd.in/ei-65Y6g Knowledge-Aware Reasoning over Multimodal Semi-structured Tables (finding) Suyash Vardhan Mathur, Jainit Bafna, Kunal Kartik, Harshita Khandelwal, Manish Shrivastava, Vivek Gupta, Mohit Bansal, Dan Roth tldr: introduce the novel task of reasoning on multimodal tabular data, text + images in a structure (first of a kind) arxiv: https://lnkd.in/e5ycusfn   Unraveling the Truth: Do LLMs really Understand Charts? A Deep Dive into Consistency and Robustness (finding) Srija Mukhopadhyay, Adnan Qidwai, Aparna Garimella, Pritika Ramu, Vivek Gupta, Dan Roth           tldr: VLM although seem very accurate on chart, are not all robust and consistent in performance arxiv: https://lnkd.in/eFFA-sVX We also got an workshop paper, at BlockBoxNLP 2024 @EMNLP, here the details Enhancing Question Answering on Charts Through Effective Pre-training Tasks Ashim Gupta, Vivek Gupta, Shuo Zhang, Yujie He, Ning Zhang , Shalin Shah, Ph.D. tldr: models is not robust & consistent on charts, can we use pre-training on simple auxiliary tasks to enhance model robustness arxiv: https://lnkd.in/egbRHm-4

385

12 Comments

Good folks from Carnegie Mellon University's WAVLab have released XEUS - A Cross-lingual Encoder for Universal Speech 🌐🗣️ Simple speech representation is hindered by a lack of labelled data. That's where Self-supervised learning (SSL) comes in! 🔍📚 XEUS is a SoTA speech encoder trained in 4000+ languages! 🏆🌍 Before this, I did not even know that there are 4000+ languages. But the paper says this is a small dent in the world's 7000+ languages. 🌐🗺️ More details: - 577M parameters 📊 - 1M+ hours of speech training data across 4057 languages 🗣️⏳ - Uses the E-Branchformer architecture and is trained using HuBERT-style masked prediction of discrete speech tokens 🏗️🔠 - For robustness, the input speech is also augmented with acoustic noise and reverberation 🎙️🔊 Beats both MMS 1B and w2v-BERT v2 2.0 🥇🏆 It requires fine-tuning to be used in downstream tasks such as Speech Recognition or Translation. 🛠️🌐 Its hidden states can also be used with k-means for semantic Speech Tokenization. 🗣️🔑 Release: https://lnkd.in/gqrfvsRF

11

I remember when this all started. Faris - invited me to catch up at our favorite spot around Blok M. I had no idea he’d just wrapped up a procedure at Pondok Indah Hospital! (He really knows how to add drama to our catch-ups.) We’d been on separate paths and projects since "the winding down project", but there he was, reminding me of our dream to ride this AI wave together again. It was a plan we’d hatched with Sahal Zain & Gurnoor Dhillon when the world of AI was still mostly theoretical and jargon-heavy. Somehow, the conversation continued with Gurnoor Dhillon while he was waiting for his meeting with someone who would eventually become one of CalvinBall’s key early adopters. It felt like everything was coming together, like a series of seemingly random dots connecting to form a bigger picture. Seeing how far we’ve come as a team at CalvinBall Technologies is truly exciting. What started as theoretical discussions has now evolved into practical Gen AI solutions that are solving everyday problems and hold endless possibilities to shape the industry. 👨‍🚀 👩‍🚀 Here’s to pushing boundaries and achieving new milestones together! Gurnoor Dhillon, Sandeep Ramesh, Faris -, Sahal Zain, Eva Zuliya, Ratu Nadiah Khairunnisa 🚀 #GenAI #AIInnovation #TechJourney #ArtificialIntelligence #TechTransformation #CalvinballTech #Calvinballers

55

2 Comments

🚀 Making LLMs work for you - A Guide to Become 10x Programmer/MLE/DS I am sure many of you use LLMs for your day-to-day work. Mostly, I have seen people use LLMs while writing code (Copilot), or as an alternative to Copilot to catch bugs, but they have become much more useful than that, provided you know how to use them well. Here are a few ways I have been using LLMs in my daily work that are helping me increase my productivity: 📚 1. Quick Paper Assessment with ELI5 Summaries - Paper abstracts are called abstracts for a reason. Asking an LLM for an ELI5 summary makes it much easier to know if I want to read a paper. 💻 2. Implementing Research Papers in Code - Try giving an LLM like Claude/ChatGPT a research paper and ask to code it. You would be surprised. Tip: Claude performs much better for this task. 🔄 3. Upgrading Code Structure and Readability - Do you want to add typing hints when writing your Python code, or do you need comments in your code, or do you have a big function that you want to be converted to classes in your code? LLMs can do it all, and you would be amazed at the code-conversion power they have. ⚡ 4. Optimizing Code Performance - The thing with LLMs is that if you tell them what to do, they can do it well. For example, if you give them a non-optimized piece of code that runs correctly, they will be able to write it in a much-optimized way. For example, I just used an LLM to convert a matrix multiplication in PyTorch to sparse MM as I was getting an OOM memory error because one of the matrices became too large. I am not even joking, but that would have taken at least 1-2 hours for me to come up with. 📝 5. Automated Documentation - Got bored of writing documentation for what you did but have the code? LLMs also work the other way around. Want to write a summary of what you did in a week, you can use an LLM for that. Saves so much time that gets wasted in formatting. 🧪 6. Streamlined Test Framework Creation - Let LLMs handle the boilerplate for Unit tests & Integration tests while you focus on core logic. 🌐 7. Language-Agnostic Development - You can paste a function in Python and ask for its equivalent in Rust, complete with language-specific best practices and error handling. This dramatically reduces the learning curve when working with new languages and helps maintain productivity during language transitions. 🤝 Apart from all of the above, LLMs work well as mentors as well as interns. LLMs are great when you just need someone to bounce ideas with. I sometimes just talk to an LLM as I would to a colleague talking about the problems I have and how I can solve them. And they come up with the right ideas or send me in the right direction a lot of times. They also act as a personal assistant for free, who can deal with writing messages, creating charts, and doing all sorts of uninteresting work for you. 💭 What's your favorite way to use LLMs? Let me know in the comments! #MachineLearning #AI #Programming

47

How do language models comprehend gibberish inputs? Our recent work with James Zou focuses on understanding the mechanisms by which LLMs can be manipulated into responding with coherent target text to seemingly gibberish inputs. Paper: https://lnkd.in/gA9Mjqc4 A few takeaways: In this work we show the prevalence of nonsensical prompts that induce LLMs to generate specific and coherent responses, which we call LM Babel. We examine the structure of Babel prompts and find that despite their high perplexity, these prompts often contain nontrivial trigger tokens, maintain lower entropy compared to random token strings, and cluster together in the model representation space. We find that the efficiency of these prompts largely depends on the prompt length as well as target text’s length and perplexity. We show that reproducing harmful texts with aligned models is not only feasible but, in some cases, even easier compared to benign texts, while fine-tuning language models to forget specific information complicates directing them towards unlearned content.

31

The next guest speaker in my LLM course is Ram Sriharsha, the CTO of Pinecone, who will talk about Retrieval Augmented Generation (RAG) on Monday October 28. ➡️ Register here: https://lu.ma/r7yokyvc Title: Knowledgeable Models Combine Retrieval with Generation: An Introduction to RAG Date: Monday October 28 from 1:45 PM to 3:15 PM ET Abstract: Large Language Models have emerged into mainstream over the last few years. While they have demonstrated some huge successes already, they have also had challenges with hallucinations, difficulty with dealing with new knowledge and in retrieving relevant, accurate knowledge for downstream tasks. In this talk, we will provide a brief overview of the problem, and discuss a promising approach to solving these problems through what is known as Retrieval Augmented Generation (RAG). Alongside, we will also introduce vector databases, what makes them different from traditional databases and how they fit into RAG. Bio: Dr. Ram Sriharsha is the CTO at Pinecone. At Pinecone, Ram is responsible for overseeing the company’s technical direction in both engineering and research. Prior to joining Pinecone, Ram held numerous senior positions at some of the most well-known companies in the tech industry. Most recently, he was Vice President of Engineering at Splunk. Before that, he was a product and engineering lead at Databricks for their unified analytics platform for genomics. Some years after receiving a PhD in Physics from the University of Maryland, Ram was a Principal Software Engineer and later a Principal Research Scientist at Yahoo. Supplementary Readings: ➡️ The original paper on RAG: "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (https://lnkd.in/e_e56Y54) ➡️ Long context considerations: "Lost in the Middle: How Language Models Use Long Contexts" (https://lnkd.in/ehED53Fe) ➡️ Article on contextual RAG from Anthropic AI (https://lnkd.in/ehK4GwMc)

69

Can we parallelise model training like transformers yet scale linearly to long sequence length just like RNN? 🤔 Yup, its possible with SSMs, infact they can do inference at computation cost of O(1). 🤯 First of all, how does it even compare with the most advanced Transformer capabilities like, - Self-Attention Mechanisms - Multi-Head Attention - Quadratic Scaling, etc. All of these have been outperformed on the new benchmarks across various domains such as language, audio, and genomics by Mamba. 🏆 Your model might be excelling with the above methods, but does it handle million-length sequences efficiently? Or process long sequences without breaking a sweat? Mamba efficiently integrates a selective state space model. What it effectively does for sequence processing is: ⚡ - Dynamically selects relevant information while discarding irrelevant data. - Operates with linear time complexity, ensuring fast and efficient processing. - Utilizes a hardware-aware parallel algorithm that maximizes GPU efficiency. - Incorporates simplified end-to-end neural network architecture without attention or MLP blocks. And during sequence modeling, - The input is processed through a selective mechanism, allowing the model to focus on the most relevant parts. - The state space parameters (such as Δ, B, and C) are dynamically adjusted based on the input, enhancing content-aware processing. - Efficient recurrent mode computation ensures faster training and inference with lower memory usage. But this approach doesn't just stop at sequence processing; it extends to real-world applications across different domains. For example: - Language modeling with Mamba outperforms Transformers of similar and even larger sizes in both pretraining and downstream evaluations. - Audio modeling with Mamba provides better quality and efficiency, making strides in speech recognition and generation. - Genomic sequence analysis with Mamba achieves state-of-the-art performance, handling long DNA sequences with ease. Mamba is not just an architectural improvement; it's a revolution in sequence modeling. I'll link the research paper down in comments, and you can explore its capabilities further. For more such detailed explanations, follow me. See ya! . . . #Mamba #SequenceModeling #AI #NeuralNetworks #Efficiency #StateSpaceModel #LanguageModeling #AudioModeling #Genomics #DeepLearning

24

📢 #NewPaperAlert Can explicitly modeling group interactions reduce spurious correlations in graph explanations? We're thrilled to share our new preprint, "Higher Order Structures For Graph Explanations," with 🚀 Akshit Sinha*, Sreeram Vennam*, & Charu Sharma In many real-world applications, post-hoc graph explainers tend to recognize spurious correlations. We hypothesize that this issue arises because traditional graphs only model pairwise interactions. To tackle this problem, we introduce FORGE, a method that incorporates higher-order structures to explicitly model group interactions. FORGE utilizes cell complexes from algebraic topology to model these higher-order interactions. The process involves two steps: 1) lifting the graph into the domain of cell complexes, and 2) transferring the importance scores back to the original graph. By adding these group interactions, we significantly improve the accuracy of graph explanations. Our work highlights the importance of explicitly modeling higher order structures for tasks requiring interpretability. 📜 Preprint: https://lnkd.in/dwhB7yMJ 🐙 GitHub: https://lnkd.in/dbUcAqcR Findings 🧵 on X: https://lnkd.in/dNSy9e5F

153

4 Comments

Google’s DeepMind has released a new method called JEST, which claims to train AI models 13 times faster and 10 times more power-efficiently than current methods. An AI model is only as good as the data it is trained on, and the time taken to train an AI model is proportional to the quantity of data. Naturally, since the new method JEST is 13 times faster in training, it has been trained on less data — which is of higher quality. The paper released by Google discusses optimizing the data selected for training the model. I found the solution to be quite smart: identify dependencies between data points and accelerate the training process by focusing on data points that match or have dependencies. Quality >> Quantity. AI systems demand a lot of energy and water for cooling today (annually ~ 4.3 GW, 0.5 litre water per 5-50 ChatGPT prompts) — JEST is a promising and positive sign for the AI world we are heading towards. #ai #deepmind #google

65

2 Comments

Facebook just released a very different kind of LLMs. These are focused on code optimization and compiler optimization. This is super important because it can fundamentally change the way code is executed on any machine. Here's how it works right now: Programs written in many languages like C/C++ have intermediate steps before they are run on the computer. For example they are converted into Byte Code and then converted into machine code via LLVM (Low level virtual machine). But Facebook's new model can take in the byte code as input and convert it to into a more efficient representation so that when this new code is converted into machine code, it will be faster. "In theory", it could mean that all existing code might be made faster without any change by developers. Our games can have higher frame rates on the same GPUs. Photoshop might run faster on the same laptop. Our software might complete tasks faster without any changes at all. But that will probably take quite a long time. To be honest I don't expect these models to actually be used in production since unless they were 100% as good as compilers they will probably break the code. And are they 100% good? No, not even close. But the speed at which improvements are happening, they will probably reach that level eventually. And that's when we will see models like these everywhere. 

11

Prof. Rao's post is a good starting point for LLM practitioners to become familiar with classic (i.e., predating LLMs) AI concepts and theories, for two reasons: 📌 To understand that some things are theoretically beyond the reach of LLMs and not waste resources chasing them. 📌 To hedge against the possibility that LLMs may be replaced by other technologies and/or undergo fundamental technological revamps, which would render all existing investments in LLMs obsolete. #artificialintelligence #machinelearning #deeplearning

41

3 Comments

Shashank Sonkar and colleagues are back with another great paper and matching GitHub repo, "Pedagogical Alignment of Large Language Models". You can read the paper on arXiv here - https://lnkd.in/enZfCJgi, and browse the code here - https://lnkd.in/gyW5hSzp. They demonstrate an interesting approach to generating synthetic data to train open weights models to behave more pedagogically, as well as developing some new evaluation metrics. I think there's lots here for those of us interested in this kind of work to learn from and build on. (If synthetic data generation is new do you, check out this post from answer.ai earlier this week - https://lnkd.in/ecsG7R4b.)

38

4 Comments

Hello everyone! I am excited to share my good fortune to be able to attend the NVIDIA AI Summit at Jio World Convention Center, BKC, Mumbai from October 24th to October 25th. The first day of the sessions, October 24th, started with a talk by NVIDIA CEO Jensen Huang, where he discussed some concepts of GPU and how it is different from the CPU, as well as the advancements NVIDIA has made in its development. Later in the session was a conversation between him and RIL Chaiman Mukesh Ambani, wherein the potential for AI in India was discussed. Jensen also shared why he was inspired to name his company "NVIDIA", which sounds similar to "INDIA". The most important moment, in my opinion, was when Jensen said that, paraphrasing, "AI will not take your job. The person who uses AI to become more productive will". In my opinion, that was the peak moment of the session. The next session I attended was by one of the co-authors of the Attention Is All You Need paper, Niki Parmar. She talked about the concept of Transformers and developments which have happened in the field. The next session was a panel talk by five industry leaders, who discussed how various enterprises develop their custom Generative AI solutions, from PoC to Production. The next session was from Tata Communications, presented by Pramod Bhandiwad, Head of Technology and Platform Development and Neelakantan Venkataraman, VP and Global Head of Cloud and Edge Business. They shed insights on how AI is integrated with Cloud Computing to deliver unparalleled AI performance, by leveraging NVIDIA GPU's, robust network and MLOps capabilities accross training, fine-tuning and deployment. It was a first-of-its-kind session for me as I learnt how AI is being used specifically by Tata, step-by-step, to achieve results. The next session was by Ganesh Sankaralingam and Theagarajan Thanikachalam, the Director and Associate Director of Latentview Analytics. The gave detailed, in-depth insights into how they developed a Generative AI solution which is revilutionising Marketing Analytics. This session is my favorite, as a lot of technical details were covered which gave a deeper understanding as to how a Generative AI solution is built from scratch, from concept to development to testing to deployment. They even shared detailed diagrams for each step, as well as their model architecture. The final event of the was was the Exhibition. Here, I witnessed practical applications of Generative AI in the fields of language translation, fraud detection, climate prediction, and so on. It was also an opportunity for me to connect with many like-minded individuals. With that, the first day of talks came to an end. I shall be telling about the second day of sessions of the AI Summit in my next post. Stay tuned! https://lnkd.in/daPqUpYN

12

2 Comments

With Sarath Shekkizhar, we are exploring the geometric structures behind LLM reasoning capabilities. So far, our findings suggest: 1. Higher density in self-attention graphs implies higher intrinsic dimension. 2. Higher intrinsic dimension induces more expressive power/reasoning with MLPs. 3. Longer context and more attention heads can boost intrinsic dimension. This geometric lens offers new insights into LLM performance and potential improvement strategies. Early days, but we believe it is a promising direction! Stay tuned for more as we dive deeper into the geometry of AI reasoning. Read our pre-print: https://lnkd.in/gcekSWYD #AI #MachineLearning #LLM

25

Quantized Side Tuning: Fast and Memory-Efficient Tuning of Quantized Large Language Models: There are two existing approaches to fine-tuning: 1. Parameter efficient fine-tuning (PEFT) (LoRa, QLoRa): update a small number of parameters; the rest will freeze. 2. Reduce the memory footprint during the training phase of fine-tuning. However, PEFT needs caching intermediate activations during forward processing, which takes the same training time as full fine-tuning and can’t reduce the memory footprint. Existing methods require a considerable amount of memory for fine-tuning. Quantized side tuning (QST) quantized LLM weights into 4-bit to reduce the memory footprint. Other models need backpropagation where QST utilizes the hidden state of LLM as a task-specific using a side network separated from LLM. QST uses several low-rank adaptors, and gradient-free downsample modules to reduce trainable parameters. https://lnkd.in/gjpK2s7w

7