Shalin Shah, Ph.D.’s Post

EMNLP 2024 results are out, we got in three papers, congrats to students :) First paper of many, Vatsal, Pranshu, Suyash, Janit, Kunal, Srija and Adnan, Congrats 🎉 🎊 🍾 🎈 Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets (main) - Vatsal Gupta, PRANSHU PANDYA, Tushar Kataria, Vivek Gupta, Dan Roth tldr: make models robust on multiple adversarial challenge set simultaneously  arxiv: https://lnkd.in/ei-65Y6g Knowledge-Aware Reasoning over Multimodal Semi-structured Tables (finding) Suyash Vardhan Mathur, Jainit Bafna, Kunal Kartik, Harshita Khandelwal, Manish Shrivastava, Vivek Gupta, Mohit Bansal, Dan Roth tldr: introduce the novel task of reasoning on multimodal tabular data, text + images in a structure (first of a kind) arxiv: https://lnkd.in/e5ycusfn   Unraveling the Truth: Do LLMs really Understand Charts? A Deep Dive into Consistency and Robustness (finding) Srija Mukhopadhyay, Adnan Qidwai, Aparna Garimella, Pritika Ramu, Vivek Gupta, Dan Roth           tldr: VLM although seem very accurate on chart, are not all robust and consistent in performance arxiv: https://lnkd.in/eFFA-sVX We also got an workshop paper, at BlockBoxNLP 2024 @EMNLP, here the details Enhancing Question Answering on Charts Through Effective Pre-training Tasks Ashim Gupta, Vivek Gupta, Shuo Zhang, Yujie He, Ning Zhang , Shalin Shah, Ph.D. tldr: models is not robust & consistent on charts, can we use pre-training on simple auxiliary tasks to enhance model robustness arxiv: https://lnkd.in/egbRHm-4

This 𝗮𝗴𝗲𝗻𝘁𝗶𝗰 𝗮𝗽𝗽𝗿𝗼𝗮𝗰𝗵 to 𝗥𝗔𝗚 leverages a 𝗴𝗿𝗮𝗽𝗵-𝗯𝗮𝘀𝗲𝗱 method with a robust data topology to enhance the precision of information retrieval. Knowledge Graphs enable 𝘀𝗲𝗮𝗿𝗰𝗵𝗶𝗻𝗴 𝗳𝗼𝗿 𝘁𝗵𝗶𝗻𝗴𝘀 𝗮𝗻𝗱 𝗻𝗼𝘁 𝘀𝘁𝗿𝗶𝗻𝗴𝘀 by maintaining extensive collections of explicit facts structured as accurate, mutable, and interpretable knowledge triples. It employs a 𝗺𝘂𝗹𝘁𝗶-𝘀𝘁𝗮𝗴𝗲 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗽𝗿𝗼𝗰𝗲𝘀𝘀, integrating a self-assessment mechanism to ensure accuracy in responses. Domain-specific queries are handled using Knowledge Graphs, while web-retrieved data is processed through parsing and chunking techniques. 𝗦𝗲𝘃𝗲𝗻 𝗞𝗲𝘆 𝗖𝗼𝗻𝘀𝗶𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀: 1. This RAG implementation is again another good example of where an Agentic Approach is followed to create a resilient, knowledge intensive conversational user interface. 2. There is also an emergence of a Graph approach from two perspectives. A graph approach is followed for agentic flows; LangChain with LangGraph, LlamaIndex Workflows and Haystack Studio from Deepset to name a few. 3. There is also a Graph approach to knowledge, where more time is spend on data discovery and data design to create stronger data topologies. 4. WeKnow-RAG is also a multi-stage approach to RAG and it is in keeping with recent developments where complexity is introduced in order to create more resilient and multifaceted solutions. 5. This approach from WeKnow-RAG includes a self-assessment mechanism. 6. KG was used for domain specific queries, while parsing and chunking was used for web retrieved data. 7. This implementation sees both Knowledge Base and a RAG/Chunking approach being combined for instances where data design is possible, or not possible. Often technologies are pitted against each other, in a one or the other scenario. Here it is illustrated that sometimes the answer is somewhere in the middle. Follow DeepNeuralAI #RAG #AI #MachineLearning #KnowledgeGraphs #NLP #AIResearch #LLMs

🎯 𝐑𝐀𝐆 𝐖𝐨𝐫𝐤𝐟𝐥𝐨𝐰 𝐄𝐱𝐩𝐥𝐚𝐢𝐧𝐞𝐝 Here’s a simplified Steps: [1] 𝑻𝒆𝒙𝒕 𝑪𝒉𝒖𝒏𝒌𝒊𝒏𝒈: Split your dataset into smaller, manageable chunks. [2] 𝑬𝒎𝒃𝒆𝒅𝒅𝒊𝒏𝒈 𝑪𝒓𝒆𝒂𝒕𝒊𝒐𝒏: Convert these text chunks into embeddings using an embedding model. [3] 𝑽𝒆𝒄𝒕𝒐𝒓 𝑺𝒕𝒐𝒓𝒂𝒈𝒆: Store these embeddings in a vector database for efficient access. [4] 𝑻𝒆𝒙𝒕 𝑴𝒂𝒑𝒑𝒊𝒏𝒈: Maintain a mapping of text chunks with their corresponding embeddings for retrieval. [5] 𝑸𝒖𝒆𝒓𝒚 𝑬𝒎𝒃𝒆𝒅𝒅𝒊𝒏𝒈: Transform user queries into embeddings using the same model as above. [6] 𝑫𝒂𝒕𝒂𝒃𝒂𝒔𝒆 𝑸𝒖𝒆𝒓𝒚𝒊𝒏𝒈: Retrieve relevant embeddings from the vector database using the query embedding. [7] 𝑺𝒊𝒎𝒊𝒍𝒂𝒓𝒊𝒕𝒚 𝑺𝒆𝒂𝒓𝒄𝒉: Identify the closest matching text chunks using approximate nearest neighbor (ANN) search. [8] 𝑪𝒐𝒏𝒕𝒆𝒙𝒕 𝑴𝒂𝒑𝒑𝒊𝒏𝒈: Map the retrieved embeddings back to their respective text chunks. [9] 𝑳𝑳𝑴 𝑹𝒆𝒔𝒑𝒐𝒏𝒔𝒆: Feed the retrieved context alongside the query into an LLM, ensuring the response remains grounded in the provided data. 𝑾𝒂𝒏𝒕 𝒕𝒐 𝒄𝒐𝒏𝒏𝒆𝒄𝒕 𝒘𝒊𝒕𝒉 𝒎𝒆? 𝘍𝒊𝒏𝒅 𝒎𝒆 𝒉𝒆𝒓𝒆 --> https://lnkd.in/dTK-FtG3 Follow Shreya Khandelwal for more such content. ************************************************************************ #DataScience #artificialintelligence #MachineLearning #LargeLanguageModels #LLMs #GenerativeAI #LanguageModels #NLP #RAG #DataEngineering #Analytics #VectorDB #Embedding

This 𝗮𝗴𝗲𝗻𝘁𝗶𝗰 𝗮𝗽𝗽𝗿𝗼𝗮𝗰𝗵 to 𝗥𝗔𝗚 leverages a 𝗴𝗿𝗮𝗽𝗵-𝗯𝗮𝘀𝗲𝗱 method with a robust data topology to enhance the precision of information retrieval. Knowledge Graphs enable 𝘀𝗲𝗮𝗿𝗰𝗵𝗶𝗻𝗴 𝗳𝗼𝗿 𝘁𝗵𝗶𝗻𝗴𝘀 𝗮𝗻𝗱 𝗻𝗼𝘁 𝘀𝘁𝗿𝗶𝗻𝗴𝘀 by maintaining extensive collections of explicit facts structured as accurate, mutable, and interpretable knowledge triples. It employs a 𝗺𝘂𝗹𝘁𝗶-𝘀𝘁𝗮𝗴𝗲 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗽𝗿𝗼𝗰𝗲𝘀𝘀, integrating a self-assessment mechanism to ensure accuracy in responses. Domain-specific queries are handled using Knowledge Graphs, while web-retrieved data is processed through parsing and chunking techniques. 𝗦𝗲𝘃𝗲𝗻 𝗞𝗲𝘆 𝗖𝗼𝗻𝘀𝗶𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀: 1. This RAG implementation is again another good example of where an Agentic Approach is followed to create a resilient, knowledge intensive conversational user interface. 2. There is also an emergence of a Graph approach from two perspectives. A graph approach is followed for agentic flows; LangChain with LangGraph, LlamaIndex Workflows and Haystack Studio from Deepset to name a few. 3. There is also a Graph approach to knowledge, where more time is spend on data discovery and data design to create stronger data topologies. 4. WeKnow-RAG is also a multi-stage approach to RAG and it is in keeping with recent developments where complexity is introduced in order to create more resilient and multifaceted solutions. 5. This approach from WeKnow-RAG includes a self-assessment mechanism. 6. KG was used for domain specific queries, while parsing and chunking was used for web retrieved data. 7. This implementation sees both Knowledge Base and a RAG/Chunking approach being combined for instances where data design is possible, or not possible. Often technologies are pitted against each other, in a one or the other scenario. Here it is illustrated that sometimes the answer is somewhere in the middle. Connect With Saif Modan #RAG #AI #MachineLearning #KnowledgeGraphs #NLP #AIResearch #LLMs

🔝 Top LLM and RAG Evaluation Frameworks 📊 Discover the best frameworks for evaluating Language Model (LLM) and Retrieval-Augmented Generation (RAG) systems. Each framework offers unique features and capabilities, catering to different aspects of LLM evaluation. 🔸 DeepEval DeepEval provides 14+ evaluation metrics for RAG and fine-tuning use cases. It integrates with Pytest and offers a hosted platform for real-time evaluations. Learn more: https://lnkd.in/e4asCZU7 🔸 MLFlow LLM Evaluate MLFlow is a modular package for running evaluations in custom pipelines, praised for its intuitive developer experience. Check it out: https://lnkd.in/eqJwFJyv 🔸 RAGAs RAGAs is designed for RAG pipelines, offering 5 core metrics. It's noted for its simplicity but has limited features compared to other frameworks. Explore RAGAs: https://lnkd.in/gQcR8ShQ 🔸 Deepchecks Deepchecks focuses on evaluating the LLM itself, known for its dashboards and visualization UI. However, it has a complicated developer experience. Discover Deepchecks: https://lnkd.in/e_9hcW-Z 🔸 Arize AI Phoenix Arize AI Phoenix evaluates LLM applications through extensive observability, offering three evaluation criteria. Learn more: https://lnkd.in/g7QHgbsY 🔸 OpenAI Evals A framework for evaluating LLMs and systems built using LLMs, offering a registry of benchmarks and the ability to write custom evals. Check it out: https://lnkd.in/efBmzmyx 🔸 TruLens TruLens focuses on evaluating and tracking LLM experiments, providing feedback functions for quality evaluation. Explore TruLens: https://lnkd.in/eqTmxxcq 📌 Image Source: https://lnkd.in/epMwvZ-v Which LLM evaluation framework do you prefer? Share your thoughts in the comments! #LLM #RAG #evaluation #frameworks #AI #NLP #LanguageModel #RetrievalAugmentedGeneration #DeepEval #MLFlow #RAGAs #Deepchecks #ArizeAIPhoenix #OpenAIEvals #TruLens #ArtificialIntelligence #NaturalLanguageProcessing #MachineLearning #EvaluationFrameworks #DataScience #DeepLearning #ModelEvaluation #AISystems #AIPipelines

Retrieval-Augmented Generation (RAG) is a technique that enhances Large Language Models (LLMs) by retrieving relevant information from external databases during the response generation process. This approach improves the accuracy and relevance of LLM responses, especially for tasks requiring specific or up-to-date knowledge. Original RAG models combine a pre-trained language model with an external knowledge source, enhancing accuracy and specificity. Graph RAG organizes data into a graph structure, handling query-focused summarization at scale. LongRAG processes larger text units, reducing the number of units to search through and enhancing performance. Self-RAG allows the model to retrieve and reflect on information only when needed, outperforming other models in reasoning and fact-checking tasks. Corrective RAG uses an external retrieval evaluator to refine the quality of retrieved documents, enhancing accuracy and robustness. EfficientRAG efficiently handles multi-hop questions by generating new queries and filtering out irrelevant information. Adaptive RAG assesses the conversation context and decides if RAG is necessary, improving response quality. Modular RAG breaks down complex RAG systems into independent modules, offering flexible configurations. Speculative RAG combines a smaller LM for producing drafts and a larger LM for verification, enhancing effectiveness and speed. RankRAG trains the model to rank relevant contexts and use them to answer questions, excelling at knowledge-intensive tasks. Multi-Head RAG uses different parts of the model’s attention mechanism to capture various aspects of a query, improving retrieval accuracy for complex queries. Types of RAG: ▪️ Original RAG ▪️ Graph RAG ▪️ LongRAG ▪️ Self-RAG ▪️ Corrective RAG ▪️ EfficientRAG ▪️ Golden-Retriever ▪️ Adaptive RAG ▪️ Modular RAG ▪️ Speculative RAG ▪️ RankRAG ▪️ Multi-Head RAG #RankRAG #RAG #LLM #NLP #multihead #LongRAG #selfRAG

Navigating Challenges with LLMs: Question Answering Over Tables When we, as humans, read a table from an article, we have the advantage of knowing the exact location of each text element. We can easily map cells of the table to their respective rows and columns. This location information (or spatial information) is not available to LLMs when we provide them with text input. While newline characters can differentiate between rows, columns present a more significant challenge. For this demonstration, I specifically selected Table 6 from the Llama3 paper (https://lnkd.in/ga7ngwUJ) because it has multi-line rows, which highlight the issue of maintaining spatial information. I asked a simple query to give value from a cell by specifying a row and column. Here's a summary of the findings: 📍 Raw Input: Simply copy-pasting the table from the PDF results in the loss of most spatial information, but it's the most straightforward way to parse any PDF programmatically. The query response was incorrect in this case. 📍 Markdown Processed: Converting the table into markdown format before querying yielded the correct answer. This method incorporated spatial knowledge for both rows and columns through a specific format. However, it is difficult to achieve programmatically as it might require manual efforts to convert tables to markdown. 📍 HTML Parsed: This method provided the correct answer and can be processed automatically without manual intervention. HTML incorporates spatial information by providing the actual coordinates of text elements in the document, which is sent as input to the LLM. This approach can be useful not only for querying tables but also for any artifacts where spatial information is important, such as graphs, charts, etc. Attaching chat links of ChatGPT for the above experiment for reference: Raw Input: https://lnkd.in/gHuJ8-7k Markdown Processed: https://lnkd.in/gvwe8Wzt HTML Parsed: https://lnkd.in/gM7EmVby #ArtificialIntelligence #MachineLearning #NLP #LLMs #DataScience #AI #Research

One line of code can reduce your vector database memory requirements by 32X and speed up retrieval by 45x with a mere 4% drop in accuracy! Embedding Quantization is a breakthrough in drastically reducing vector database index size and retrieval latency with minimal accuracy loss! Embedding Quantization is easy with Sentence Transformers: 1. Pick a suitable Sentence Transformer model, like "mixedbread-ai/mxbai-embed-large-v1," which is currently a leading model in the MTEB benchmark. 2. Generate the sentence embeddings as usual. This sentence embedding is a 1024 dimension array, each being a floating-point value. 3. [NEW] Add one line of code to convert the regular embeddings to binary quantized embeddings. Each floating point value in the sentence embedding array is converted to binary with one if the float value exceeds 0. Now that we have 1024 binary values, we can compress eight binary numbers into a single integer; hence, the dimensionality drops from 1024 to 128. 4. Finally, we use hamming distance instead of cosine similarity to compare the sentences using these integer arrays. Hamming distance is equivalent to cosine distance for binary vectors but massively faster to compute and with the added advantage of allowing dimensionality compression from 1024 to 128! I tested the "mixedbread-ai/mxbai-embed-large-v1" and everyone's favorite "all-MiniLM-L6-v2" models with excellent results, so the method seems to work on multiple sentence transformer models. Most vector DBs, such as Qdrant and The Milvus Project, already support binary embeddings, so using them in your RAG applications is straightforward to improve your app's performance! Sadly, I couldn't find any information on support in Pinecone. 🤗 Detailed blog post on Hugging Face: https://lnkd.in/eKaQNbEm Follow me for more tips on building successful ML and LLM products! Medium: https://lnkd.in/g2jAJn5 X: https://lnkd.in/g_JbKEkM #generativeai #llm #nlp #artificialintelligence #mlops #llmops

Who ever said you don't need Mathematics to be a software engineer. This insightful post talks about some concepts that I learnt during my JEE preparation. You might not need maths for many other things in IT. But AI algorithms is not one of them. #ai #genai