Kalyan KS

𝗪𝗶𝗹𝗹 𝗔𝗜 𝗛𝗶𝘁 𝗮 𝗗𝗮𝘁𝗮 𝗪𝗮𝗹𝗹 𝗶𝗻 𝟮𝟬𝟯𝟬? 🆘 Very interesting analysis from epoch.ai about data corpus available for training large language models and whether we will hit the ceiling soon. https://lnkd.in/gD78f9DW The authors estimate that all available data will be used by ~2030, but they're also the first to admit that by then we will have ways to address "the shortage" (such as synthetic data, data efficiency improvements, use of other modalities). What I found more interesting though is why the latest predictions differ from their 2022 paper, where they speculated that all the data will be used by 2024: ☑ Web data proves more valuable than expected (on par with scientific papers) ☑ Models can "reuse" datasets multiple times 2030 is right around the corner, so we will see a lot of innovation on this front and creative ideas on how to 'recycle' and make best use of the data we already have. These estimates also don't take into account proprietary data owned by individual companies. In this No priors episode (https://lnkd.in/g2BkTJBz) there is a mention of JPMorgan owning 150PB of proprietary data, which is at least 150x of what was used to train GPT-4. JPMorgan is monetizing this data today (https://lnkd.in/gXjE92qx), but it gets exponentially more valuable in a world where data is a finite (and extremely sought after) resource.

12

Shashank Sonkar and colleagues are back with another great paper and matching GitHub repo, "Pedagogical Alignment of Large Language Models". You can read the paper on arXiv here - https://lnkd.in/enZfCJgi, and browse the code here - https://lnkd.in/gyW5hSzp. They demonstrate an interesting approach to generating synthetic data to train open weights models to behave more pedagogically, as well as developing some new evaluation metrics. I think there's lots here for those of us interested in this kind of work to learn from and build on. (If synthetic data generation is new do you, check out this post from answer.ai earlier this week - https://lnkd.in/ecsG7R4b.)

38

4 Comments

IndicGenBench Google Research India recently released IndicGenBench, a multilingual benchmark to evaluate generation capabilities of LLMs on 29 Indic languages spanning 13 writing scripts and 4 language families. Extended the datasets in Cross-lingual Summarization, Machine Translation, Multi-lingual Question Answering, and Cross-lingual Question Answering, the team has collected human translations for English examples into target Indic languages, thereby extending the scope and applicability of evaluation metrics in this domain. One of the key insights from their study is the analysis of token fertility across all Indic languages within IndicGenBench. Token fertility, representing the average number of sub-words that a word is broken down into by the tokenizer, varies significantly across languages Some languages have simple breakdowns, while others are more complex. Now, why does this matter? Well, it affects how well the language models work. Languages with more complex breakdowns might struggle because they can't use as many examples to learn from. They found that languages with simpler breakdowns can use more examples effectively compared to those with complex breakdowns. #LLMs #GenAI #IndicGenBench #AI #IndicLanguages #IndicDatasets #multilingual

15

1 Comment

Quantized embeddings offer significant advantages for NLP models, including reduced storage requirements, faster computations, lower energy consumption, and improved model interpretability. They are becoming increasingly important for enabling efficient and scalable NLP on resource-constrained devices and cloud-based applications. Read my latest article below

6

Text chunks in retrieval may risk losing context. The below article introduces contextualizing each chunk using an LLM. The added cost and latency required for contextualizing is partially addressed through a lightweight LLM (haiku) and prompt caching. Along with other known techniques (BM25, reranking, etc.), a significant reduction in retrieval failure is achieved. Overall, a good article to read. https://lnkd.in/g_5fgAN8

10

Just published a new developer walkthrough 🔥 Attention Mechanisms Simplified: Using einsum in PyTorch Implement various attention mechanisms, such as self-attention and multi-head attention, using einsum. https://buff.ly/3XtPNjW

1

Jianbo Shi and Jitendra Malik's paper on "Normalized Cuts and Image Segmentation" (https://lnkd.in/dcyU5Skr) published in 2000, is a seminal work that introduces an interesting method for image segmentation by using concepts from graph theory and linear algebra. It is one of the most highly cited papers in Computer Vision with over 20k+ citations. By modeling the image as a graph, where pixels represent nodes and the edges between them encode the similarity, the segmentation problem transforms into one of graph partitioning. The segments then can be thought of as subgraphs. The key idea is to partition the graph to minimize the normalized cut (a criterion that evaluates the cut cost: the total weight of the edges removed to split the graph) relative to the total edge connections within each group. This problem is then elegantly converted into a linear algebra problem: finding the partition corresponds to solving for the second smallest eigenvector of the Laplacian matrix of the graph. Here's a short report on the entire paper, prepared as part of our project component for the Linear Algebra and its Applications course last semester. Please feel free to comment or address any issues if there are any! #ImageSegmentation #MachineLearning #ComputerVision #DataScience #ScientificResearch #LinearAlgebra #GraphTheory #Report #project

125

5 Comments

Hello Everyone Your learning - #learnwithvanama ---- 🔍 Exploring Best Practices in Retrieval-Augmented Generation (RAG) 📄 I recently delved into the insightful paper "Searching for Best Practices in Retrieval-Augmented Generation" by Xiaohua Wang et al., which offers a comprehensive survey on RAG techniques. This innovative approach merges large language models with information retrieval, enhancing content accuracy and relevance by tapping into external sources. Key Findings: - Mitigating Hallucinations: RAG proves effective in reducing inaccuracies and improving response quality, especially in specialized fields. - Multimodal Enhancements: Techniques for multimodal retrieval enhance question-answering related to visual inputs and expedite the generation of multimodal content. - Challenges: - Implementation complexities and extended response times are notable hurdles. - Integration Potential: Combining RAG with reinforcement learning or few-shot learning could yield more robust AI systems. Best Practices: - Continuous Evaluation: Regularly assess outputs to identify areas for enhancement. - Transparency: Clearly explain how outputs are generated to build user trust. - Personalization: Use customer integration data to tailor AI features effectively. The paper serves as a significant step forward in refining retrieval-augmented generation. While challenges persist, the proposed strategies and best practices offer a solid foundation for optimizing RAG and advancing AI capabilities. Embracing these insights can drive the development of more knowledgeable and contextually-aware AI systems. Link : https://lnkd.in/e3YJfq4q #AI #MachineLearning #Research #DataScience #Innovation #TechTrends

22

Brain Teaser #10: Suppose we are fine-tuning a large language model using Low-Rank Adaptation (LoRA). We possess two low-rank update matrices, δWA​ and δWB​​, both associated with the rank "r". Considering our objective to preserve the model's current knowledge while incorporating new data from diverse features, what "mathematical rationale" should guide the selection of one of the update matrices (δWA​ or δWB) for low-rank adaptation? Hint: Consider the specific characteristics or properties of the matrices that can help in developing mathematical reasoning. Here is my response: The norm ratio, calculated as the ratio of the nuclear norm to the Frobenius norm of the update matrix, is an essential metric for assessing how updates impact a model. It provides insights into the model's stability and adaptability. Here’s a brief overview: https://lnkd.in/ekB_q9XX #MachineLearning #DeepLearning #NaturalLanguageProcessing #LoRA #FineTuning #AI #DataScience #ModelOptimization #NeuralNetworks #ArtificialIntelligence #TechInnovation #DataDriven #KnowledgeIntegration #Research #LLMs

30

4 Comments

Furthering research for enhanced reliability in Large Language Models (LLMs), a ground-breaking framework - AGREE (Adaptation for GRounding EnhancEment) has been developed. Empowering LLMs with improved grounding and citation precision, AGREE eases out the hallucination issue - nonsubstantiated plausible information generated by LLMs.👏✨ Comprehensive experiments oblige AGREE leading to an impressive 30% relative improvement in grounding. 🚀Continual learning with an essence of collaboration pledges user trust and myriad potential applications! Eager to dive deeper? Click on: [https://lnkd.in/dgHhRsnn Advancements shared today inspire the revelations of tomorrow! #IntelligentAutomation #AGREEFramework #LLMs 👨‍💻🤖

3

I was reading about Phi-2 as I have been interested in making large language models smaller. This is the blog that I read: https://lnkd.in/gwvikUhZ Question now is very simple, can this be the rise of so called small language models? Should we not try to make large language models smarter but with without having that mountain of redundancy of large language models?

1

For those who couldn't make it to SF in September for the AI Conference, check out the video link for the talk I gave on Graph RAG here 👇🏽. https://lnkd.in/gKR7AS4T The original BlackRock/Nvidia Hybrid RAG paper mentions triples (likely knowledge graphs using an RDF data model), and it's also light on implementation details of graph retrieval. Recently, I've been experimenting with strategies that utilize lightweight prompting frameworks and a reranker to combine results from graph + vector RAG, and this approach works surprisingly well even when using Kùzu's property graph data model and text2Cypher to retrieve from the graph. Give the video a watch first, and then check out the code here: https://lnkd.in/gaAWjqYv Happy to discuss more with anyone who has thoughts/comments on this methodology!

19

#neurips #ml #highschool Hi everyone, The top ML conference NeurIPS just called for research papers made by high school students, making waves in the ML research community! Wanna know more about the debate? Check out my YouTube video: https://lnkd.in/gphKUR2i

8

💥💥💥 LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Yuxiao Dong, Jie Tang Abstract Long-Context Question Answering (LCQA), a challenging task, aims to reason over long-context documents to yield accurate answers to questions. Existing long-context Large Language Models (LLMs) for LCQA often struggle with the "lost in the middle" issue. Retrieval-Augmented Generation (RAG) mitigates this issue by providing external factual evidence. However, its chunking strategy disrupts the global long-context information, and its low-quality retrieval in long contexts hinders LLMs from identifying effective factual details due to substantial noise. To this end, we propose LongRAG, a general, dual-perspective, and robust LLM-based RAG system paradigm for LCQA to enhance RAG's understanding of complex long-context knowledge (i.e., global information and factual details). We design LongRAG as a plug-and-play paradigm, facilitating adaptation to various domains and LLMs. Extensive experiments on three multi-hop datasets demonstrate that LongRAG significantly outperforms long-context LLMs (up by 6.94%), advanced RAG (up by 6.16%), and Vanilla RAG (up by 17.25%). Furthermore, we conduct quantitative ablation studies and multi-dimensional analyses, highlighting the effectiveness of the system's components and fine-tuning strategies. 👉 https://lnkd.in/d7PaajBQ #machinelearning

1

🔊Another Mock NLP Interview session 😎 🔗 link: Mock NLP Interview: Top Questions & Answers | Prepare for Your NLP Job Interview https://lnkd.in/d2S9u9zY 📓📕This episode of Mock Interview examines ✅ typical questions asked in technical Narural Language Processing (NLP) interviews, ✅offers a comprehensive insight into the key concepts developers should revise before facing such interviews. 👩‍🏭If you're interested in a mock interview, please direct message 🧑‍💻me as soon as possible. The first 10 interviews will be free 🆓, with a fee for the rest. Act quickly! 🥳Share your suggestions for the next tech stack you'd like me to cover by leaving a comment below. #nlp #naturallanguageprocessing #datascience #mockinterview #interviewpreparation #ai #generativeai #llm

15

1 Comment

Understanding Temperature in Language Models 🌡️ Temperature is a pivotal parameter in large language models influencing the diversity and coherence of generated text through softmax-based sampling. Temperature is a parameter used in softmax-based sampling methods within language models. In essence, softmax converts logits (raw model outputs) into probabilities. Temperature scales these probabilities before sampling from them. The formula for temperature scaling is: Scaled logits = logits / T, where T is the temperature parameter. Higher temperatures lead to more uniform distributions over the vocabulary, encouraging the model to explore diverse word choices. Conversely, lower temperatures sharpen the distribution, favoring high-confidence predictions and reducing diversity in generated text. Diversity vs. Coherence: High Temperature: Increases the entropy of the probability distribution, encouraging the model to explore a wider range of word choices. This enhances creativity in generated text. Low Temperature: Decreases entropy, favoring high-confidence predictions and reducing randomness. This improves coherence and ensures more predictable outputs. Understanding temperature empowers researchers and developers to optimize language model outputs for specific applications, striking a balance between creativity and coherence. Let's delve deeper into the nuances of AI and NLP together! 💡 #NLP #AI #MachineLearning #DataScience #NaturalLanguageProcessing #LLMs Feel free to connect and share your experiences or ask any questions! 🙌

3