Dhruv Batra

EMNLP 2024 results are out, we got in three papers, congrats to students :) First paper of many, Vatsal, Pranshu, Suyash, Janit, Kunal, Srija and Adnan, Congrats 🎉 🎊 🍾 🎈 Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets (main) - Vatsal Gupta, PRANSHU PANDYA, Tushar Kataria, Vivek Gupta, Dan Roth tldr: make models robust on multiple adversarial challenge set simultaneously  arxiv: https://lnkd.in/ei-65Y6g Knowledge-Aware Reasoning over Multimodal Semi-structured Tables (finding) Suyash Vardhan Mathur, Jainit Bafna, Kunal Kartik, Harshita Khandelwal, Manish Shrivastava, Vivek Gupta, Mohit Bansal, Dan Roth tldr: introduce the novel task of reasoning on multimodal tabular data, text + images in a structure (first of a kind) arxiv: https://lnkd.in/e5ycusfn   Unraveling the Truth: Do LLMs really Understand Charts? A Deep Dive into Consistency and Robustness (finding) Srija Mukhopadhyay, Adnan Qidwai, Aparna Garimella, Pritika Ramu, Vivek Gupta, Dan Roth           tldr: VLM although seem very accurate on chart, are not all robust and consistent in performance arxiv: https://lnkd.in/eFFA-sVX We also got an workshop paper, at BlockBoxNLP 2024 @EMNLP, here the details Enhancing Question Answering on Charts Through Effective Pre-training Tasks Ashim Gupta, Vivek Gupta, Shuo Zhang, Yujie He, Ning Zhang , Shalin Shah, Ph.D. tldr: models is not robust & consistent on charts, can we use pre-training on simple auxiliary tasks to enhance model robustness arxiv: https://lnkd.in/egbRHm-4

385

12 Comments

📢 #NewPaperAlert Can explicitly modeling group interactions reduce spurious correlations in graph explanations? We're thrilled to share our new preprint, "Higher Order Structures For Graph Explanations," with 🚀 Akshit Sinha*, Sreeram Vennam*, & Charu Sharma In many real-world applications, post-hoc graph explainers tend to recognize spurious correlations. We hypothesize that this issue arises because traditional graphs only model pairwise interactions. To tackle this problem, we introduce FORGE, a method that incorporates higher-order structures to explicitly model group interactions. FORGE utilizes cell complexes from algebraic topology to model these higher-order interactions. The process involves two steps: 1) lifting the graph into the domain of cell complexes, and 2) transferring the importance scores back to the original graph. By adding these group interactions, we significantly improve the accuracy of graph explanations. Our work highlights the importance of explicitly modeling higher order structures for tasks requiring interpretability. 📜 Preprint: https://lnkd.in/dwhB7yMJ 🐙 GitHub: https://lnkd.in/dbUcAqcR Findings 🧵 on X: https://lnkd.in/dNSy9e5F

153

4 Comments

The next guest speaker in my LLM course is Ram Sriharsha, the CTO of Pinecone, who will talk about Retrieval Augmented Generation (RAG) on Monday October 28. ➡️ Register here: https://lu.ma/r7yokyvc Title: Knowledgeable Models Combine Retrieval with Generation: An Introduction to RAG Date: Monday October 28 from 1:45 PM to 3:15 PM ET Abstract: Large Language Models have emerged into mainstream over the last few years. While they have demonstrated some huge successes already, they have also had challenges with hallucinations, difficulty with dealing with new knowledge and in retrieving relevant, accurate knowledge for downstream tasks. In this talk, we will provide a brief overview of the problem, and discuss a promising approach to solving these problems through what is known as Retrieval Augmented Generation (RAG). Alongside, we will also introduce vector databases, what makes them different from traditional databases and how they fit into RAG. Bio: Dr. Ram Sriharsha is the CTO at Pinecone. At Pinecone, Ram is responsible for overseeing the company’s technical direction in both engineering and research. Prior to joining Pinecone, Ram held numerous senior positions at some of the most well-known companies in the tech industry. Most recently, he was Vice President of Engineering at Splunk. Before that, he was a product and engineering lead at Databricks for their unified analytics platform for genomics. Some years after receiving a PhD in Physics from the University of Maryland, Ram was a Principal Software Engineer and later a Principal Research Scientist at Yahoo. Supplementary Readings: ➡️ The original paper on RAG: "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (https://lnkd.in/e_e56Y54) ➡️ Long context considerations: "Lost in the Middle: How Language Models Use Long Contexts" (https://lnkd.in/ehED53Fe) ➡️ Article on contextual RAG from Anthropic AI (https://lnkd.in/ehK4GwMc)

69

IndicGenBench Google Research India recently released IndicGenBench, a multilingual benchmark to evaluate generation capabilities of LLMs on 29 Indic languages spanning 13 writing scripts and 4 language families. Extended the datasets in Cross-lingual Summarization, Machine Translation, Multi-lingual Question Answering, and Cross-lingual Question Answering, the team has collected human translations for English examples into target Indic languages, thereby extending the scope and applicability of evaluation metrics in this domain. One of the key insights from their study is the analysis of token fertility across all Indic languages within IndicGenBench. Token fertility, representing the average number of sub-words that a word is broken down into by the tokenizer, varies significantly across languages Some languages have simple breakdowns, while others are more complex. Now, why does this matter? Well, it affects how well the language models work. Languages with more complex breakdowns might struggle because they can't use as many examples to learn from. They found that languages with simpler breakdowns can use more examples effectively compared to those with complex breakdowns. #LLMs #GenAI #IndicGenBench #AI #IndicLanguages #IndicDatasets #multilingual

15

1 Comment

A comprehensive overview and synthesis of retrieval-enhanced machine learning models (RAG and beyond)! 👇

43

🎉 Really excited to share that our paper, 'Unveiling Hallucination in Text, Image, Video, and Audio Foundation Models: A Comprehensive Review' has been accepted in #EMNLP2024 Findings! by Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, Aman Chadha 📄 Overview of the Paper: 🔹 Broad Scope: Focuses on hallucination detection and mitigation for multimodal foundation models, not just language models. 🛠️ Techniques: Presents various strategies to tackle hallucination in multimodal settings. 🚧 Challenges: Highlights open challenges and suggests future research directions in this crucial field.

117

2 Comments

What a great talk from Swapnil Sayan Saha, on Physics-Aware Tiny Machine Learning! https://lnkd.in/g9g4YWs4 There's exciting times ahead as Physics-aware TinyML is becoming increasingly important in merging the power of machine learning with operating in the physical world. By integrating real-world physics models into ultra-low-power devices, this approach dramatically enhances the accuracy and efficiency of TinyML applications. From improving energy efficiency in smart sensors to enabling more accurate anomaly detection in industrial applications, this approach to machine learning at the edge helps machines better understand their environment - without sacrificing power or performance. As we explore the possibilities, this combination of physics and Edge AI opens up new opportunities in fields like healthcare, environmental monitoring, and autonomous systems. It's a huge leap toward making AI more sustainable and impactful, and allows us to build more useful AI-enabled systems that operate in (and on!) the physical world. #TinyML #EdgeAI #PhysicsAwareAI #AIInnovation #Sustainability #MachineLearning

4

If you are coming to EMNLP, let’s talk. Happy to chat more about the paper (anything Document Understanding and multi-modal AI) and other things. https://lnkd.in/ef9yPS6v #emnlp2024

16

No JEE? IIT ka exposure phir bhi!😉 In a recent video, Nova revealed how to get into cutting-edge AI and Data Science programs without even cracking JEE! 🤯 He also shared some cool ways to level up and grab your dream job with an IIT degree! Here are some highlights: ✅ Waiver from entrance exam & mentorship from industry leaders ✅ Hands-on training with 100+ AI tools & technologies ✅ Both online & on-campus learning formats Curious to know more? Click the link below to watch the full video!✨ . . . #futurense #futurenseuni #iit #career #upskill

14

1 Comment

Tanay Jaipuria’s analysis on OpenAI’s o-1 model highlights how inference-time scaling laws, allowing models to “think” longer on complex queries, could enhance performance without extra training. This marks a shift toward dynamic reasoning, where compute resources are optimized at inference rather than pre-training. As AI models tackle harder, more consequential tasks, this approach could transform both AI deployment strategies and cost structures. Are inference-time improvements the next key to AI advancement? Let’s discuss!

3

🔥 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈 𝐚𝐧𝐝 𝐋𝐋𝐌 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 - 𝐓𝐨𝐩 𝐪𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬 🔥 🤖 I've conducted a mock interview with my student, Bheesetti Anand, focused on Generative AI and LLM (Large Language Models) interview questions. This session is designed to benefit other students preparing for similar interviews, helping them secure their dream roles in the field of Generative AI. 📔 Here’s why watching the video can help you make a 𝐬𝐮𝐜𝐜𝐞𝐬𝐬𝐟𝐮𝐥 𝐜𝐚𝐫𝐞𝐞𝐫 𝐬𝐰𝐢𝐭𝐜𝐡 𝐭𝐨 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈: ✅ 𝐂𝐨𝐦𝐩𝐫𝐞𝐡𝐞𝐧𝐬𝐢𝐯𝐞 𝐂𝐨𝐯𝐞𝐫𝐚𝐠𝐞: Detailed exploration of top interview questions related to Generative AI and Large Language Models. ✅ 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐈𝐧𝐬𝐢𝐠𝐡𝐭𝐬: Real-world examples and practical applications to understand complex concepts better. ✅ 𝐄𝐱𝐩𝐞𝐫𝐭 𝐆𝐮𝐢𝐝𝐚𝐧𝐜𝐞: Tips and strategies to approach and answer interview questions effectively. ✅ 𝐂𝐚𝐫𝐞𝐞𝐫 𝐏𝐫𝐞𝐩𝐚𝐫𝐚𝐭𝐢𝐨𝐧: Essential knowledge and skills to prepare for interviews and stand out to potential employers. 📽 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 𝐥𝐢𝐧𝐤: https://lnkd.in/g_nkP7fs #generativeai #llm #interview #jobpreparation #mockinterview

14

2 Comments

🚨ALERT: Innovation in artificial neural networks🚨 Published earlier today: “SWAG: A Novel Neural Network Architecture Leveraging Polynomial Activation Functions for Enhanced Deep Learning Efficiency” Kudos to Dr. Saeid Safaei, my last doctoral student at the The University of Georgia. This work is significant because it breaks a dogma in artificial neural network (ANN) literature. In 1993, Leshno et al. reported that an ANN “… can approximate any continuous function to any degree of accuracy if and only if the network's activation function is not a polynomial”. This myth became legend, the legend became dogma. We showed that not only it is possible to use polynomials… it is desirable. Our method with simple architectures outperforms traditional ANNs with complex architectures. It could have profound implications to decrease energy demand for AI through the development of photonic chips. https://lnkd.in/gQAtcNEa

23

8 Comments

The Joy of Teaching: Learning by Teaching My first full course teaching experience - Computer Vision (CS 763) at IIT Bombay in 2018-2019 taught me the profound truth of the "Feynman Technique": explaining complex concepts as if to a child. Let me share a specific example: When preparing lectures on Convolutional Neural Networks, I had to break down every detail to its core. What exactly is convolution and how does it differ from correlation? In 2D operations, what happens to the kernel in each case? When PyTorch implements "convolution"(torch.nn.Conv2d), is it actually performing "correlation"? And the most fascinating part - why is the backpropagation of a convolution itself a convolution, and of what? I couldn't handwave explanations to myself as I might when learning alone. This experience taught me that true understanding isn't about memorizing formulas - it's about explaining complex ideas simply. As Feynman said, "If you can't explain something to a first-year student, then you haven't really understood it." Teaching didn't just make me a better instructor - it made me a better learner. I look forward to co-teaching the next DLCV course with Prof. Venkatesh Babu Radhakrishnan in Spring 2025, where I'll teach Diffusion Models for the first time. Another opportunity to learn by teaching! A special shoutout to Rishabh Dabral, who was a PhD student with me and TA for this course - now a group leader (asst. prof.) at the Max Planck Institute for Informatics. He did most of the heavy lifting, and I couldn't have done it without him. [Link to my 2019 lecture in comments] #TeachingIsLearning #ComputerVision #DeepLearning #Education

224

18 Comments

Fine tuning gpt2 on indic-gpt dataset I ran one epoch of fine tuning on an arbitrary dataset of Indic-gpt and managed to train it for language detection with Colab.Have a look: https://lnkd.in/dZJUY-UV #indic-gpt #fine tuning #language detection

10

I'm super excited to share our recent work OmniEdit, an omnipotent editing model to handle all different types of editing requests including addition, removal, swapping, environment, background, style, etc. The best part is the **highest-quality** 1.2M high-resolution image editing dataset in https://lnkd.in/g4xQePyT. The biggest blocker in image editing is the lack of high-quality editing pairs. Most existing released datasets are highly noisy, low-resolution, with strong artifacts. This basically prohibits the progress in this area. We spent **8 months** to experiment with many approaches to synthesize and filter clean image editing pairs. Eventually, we built seven specialized pipelines to propose massive amount of candidates and then prompt GPT-4o to assign quality scores these candidates. We took the highest-ranked candidates as our 1.2M training data. We compare our dataset with the existing datasets. Our dataset comes with diverse, high-resolution, any aspect ratio, and high-quality editing pairs. For more details, check out our paper: https://lnkd.in/gnS6Me37.

30

I am thrilled to announce that a paper from Elementera AI, "When Transfer Learning is Possible?" has been accepted to the [ICML] Int'l Conference on Machine Learning 2024 conference! My co-founder, Soroush Mehri, summarized the paper as follows: Our research explores when and how knowledge can be effectively transferred across different problems, a crucial step in making AI more adaptable and efficient especially for business problems. Most relevant to practitioners of AI systems, we are able to challenge widely-held beliefs about transfer learning. First, we show that having sparse changes across environments is neither necessary nor sufficient for transfer. Second, we show an example where the common heuristic of freezing a layer in a network can potentially cause poor transfer performance. In practice, many use transfer learning in the form of fine-tuning to solve business problems, such as fine-tuning LLMs and foundation models. However, our research questions the assumptions underlying this approach. Transfer learning is not as straightforward as it may seem, and there are risks that businesses might not be aware of. By challenging conventional beliefs and offering new insights, we aim to simplify AI adoption for businesses, aligning perfectly with our mission at Elementera. At Elementera AI, we aim to make AI integration seamless from model selection to post-adoption monitoring. Mark Zuckerberg at Meta today echoed that Open Source AI is the path forward, a belief that aligns perfectly with our mission. At Elementera AI, we strive to democratize AI for businesses everywhere. Poster session on Thu 25 Jul 1:30 - 3 p.m. CEST:  https://lnkd.in/eweMKE5c Paper: https://lnkd.in/eEEgm7Bn Stay tuned for more updates as we continue to innovate and lead in the field of AI! We are currently accepting requests for private testers, so please join our private tester list Elementera AI. #AI #MachineLearning #ICML2024 #TransferLearning #AIResearch #OpenSourceAI #BusinessAI #Innovation #ElementeraAI #AIAdoption

24

2 Comments

Great work here Rohan Sidankar . Thanks to Rohan and Justin, hushh is known for its work in optimizing neural network search through dimensionality reduction, specifically with ViT-B-32 CLIP for various search tasks as we build out our search index. This allows for maintaining recall while reducing vector size, enhancing search efficiency and effectiveness against large datasets like the LAION Benchmark Repository. Dimensionality reduction impacts neural network search by enabling the maintenance of recall while reducing vector size. This optimization, as seen in Rohan Sidankar's work at Hushh with ViT-B-32 CLIP, enhances search efficiency and effectiveness against large datasets like the LAION Benchmark Repository. ViT-B-32 CLIP can enhance search tasks by allowing for dimensionality reduction in neural network search, as demonstrated by Rohan Sidankar's work at Hushh. This approach enables the maintenance of recall while reducing the vector size, which is crucial for efficient and effective search tasks against large datasets like the LAION Benchmark Repository.

11