Kaushik Shakkari

🚀 Making LLMs work for you - A Guide to Become 10x Programmer/MLE/DS I am sure many of you use LLMs for your day-to-day work. Mostly, I have seen people use LLMs while writing code (Copilot), or as an alternative to Copilot to catch bugs, but they have become much more useful than that, provided you know how to use them well. Here are a few ways I have been using LLMs in my daily work that are helping me increase my productivity: 📚 1. Quick Paper Assessment with ELI5 Summaries - Paper abstracts are called abstracts for a reason. Asking an LLM for an ELI5 summary makes it much easier to know if I want to read a paper. 💻 2. Implementing Research Papers in Code - Try giving an LLM like Claude/ChatGPT a research paper and ask to code it. You would be surprised. Tip: Claude performs much better for this task. 🔄 3. Upgrading Code Structure and Readability - Do you want to add typing hints when writing your Python code, or do you need comments in your code, or do you have a big function that you want to be converted to classes in your code? LLMs can do it all, and you would be amazed at the code-conversion power they have. ⚡ 4. Optimizing Code Performance - The thing with LLMs is that if you tell them what to do, they can do it well. For example, if you give them a non-optimized piece of code that runs correctly, they will be able to write it in a much-optimized way. For example, I just used an LLM to convert a matrix multiplication in PyTorch to sparse MM as I was getting an OOM memory error because one of the matrices became too large. I am not even joking, but that would have taken at least 1-2 hours for me to come up with. 📝 5. Automated Documentation - Got bored of writing documentation for what you did but have the code? LLMs also work the other way around. Want to write a summary of what you did in a week, you can use an LLM for that. Saves so much time that gets wasted in formatting. 🧪 6. Streamlined Test Framework Creation - Let LLMs handle the boilerplate for Unit tests & Integration tests while you focus on core logic. 🌐 7. Language-Agnostic Development - You can paste a function in Python and ask for its equivalent in Rust, complete with language-specific best practices and error handling. This dramatically reduces the learning curve when working with new languages and helps maintain productivity during language transitions. 🤝 Apart from all of the above, LLMs work well as mentors as well as interns. LLMs are great when you just need someone to bounce ideas with. I sometimes just talk to an LLM as I would to a colleague talking about the problems I have and how I can solve them. And they come up with the right ideas or send me in the right direction a lot of times. They also act as a personal assistant for free, who can deal with writing messages, creating charts, and doing all sorts of uninteresting work for you. 💭 What's your favorite way to use LLMs? Let me know in the comments! #MachineLearning #AI #Programming

47

Sal Khan says “… if the technology is pencil and paper, so be it …” - guess what, this is why we have been making Smart Paper technology to integrate paper and digital! Paper is not going away and handwriting is a key component of learning activities, especially in STEM subjects. Don’t do math in the screen, do it on paper, it is way more effective. Paper is a canvas to aid and develop thinking, and with Smart Paper, we are making it easy to reliably digitize paper based formative assessments and provide AI feedback on them. Smart Paper does a little more than just sending the page photo to a VLM. We have developed it for reliability and massive mobile scalability. Smart Paper is already helping Rajasthan Government digitize and score 40+ million multiple choice worksheets. Smart worksheets have full design flexibility, going beyond traditional omr, and you can scan them with a simple photo click. It works with phones that are 7-8 years old as well, making our system scalable in most geographies and real world conditions. Now, Smart Paper has gone beyond the multiple choice and started providing feedback and grading for descriptive answered including math problems. We are working to relieve teachers from grading workloads while helping students get effective feedback from assessments. The digital data from paper can further enhance and personalise digital learning. I believe that for a long while, pencil and paper will remain important tools in classrooms and it will be most productive if we integrate paper with digital.

25

1 Comment

Meta’s new language model "Llama 3" is finally here. Plus, you can access it directly with chatGPT like interface. (not available in India yet) ► Integration with Meta AI: Llama 3 is now part of Meta AI, facilitating an array of tasks including coding and problem-solving. ► Model Variants: Available in both 8B and 70B configurations, offering versatility for different development needs. ► Enhanced Training: Trained on a 15T token dataset, 7x larger than Llama 2, with four times more code content. ► Extended Context Length: Supports an 8K context length, doubling the capacity of the previous model, Llama 2. ► Infrastructure: Utilizes Meta's recently built 24K GPU clusters, enhancing its data handling and scalability capabilities. ► Access and Usage: Users can interact with Llama 3 through a ChatGPT-like interface, enhancing user accessibility and ease of use.

4

Good folks from Carnegie Mellon University's WAVLab have released XEUS - A Cross-lingual Encoder for Universal Speech 🌐🗣️ Simple speech representation is hindered by a lack of labelled data. That's where Self-supervised learning (SSL) comes in! 🔍📚 XEUS is a SoTA speech encoder trained in 4000+ languages! 🏆🌍 Before this, I did not even know that there are 4000+ languages. But the paper says this is a small dent in the world's 7000+ languages. 🌐🗺️ More details: - 577M parameters 📊 - 1M+ hours of speech training data across 4057 languages 🗣️⏳ - Uses the E-Branchformer architecture and is trained using HuBERT-style masked prediction of discrete speech tokens 🏗️🔠 - For robustness, the input speech is also augmented with acoustic noise and reverberation 🎙️🔊 Beats both MMS 1B and w2v-BERT v2 2.0 🥇🏆 It requires fine-tuning to be used in downstream tasks such as Speech Recognition or Translation. 🛠️🌐 Its hidden states can also be used with k-means for semantic Speech Tokenization. 🗣️🔑 Release: https://lnkd.in/gqrfvsRF

11

🎓 Is a Masters in USA in Computer Sciene/Data Science still worth it? There is a lot of speculation around this question, with clickbait news coming up with random facts and suggestions. The economy and job market has been tough in US and across the world in 2024 and there is uncertainties around visa and work permit. Lots of student are yet to land an internship or job offer with surmounting pressure of visa validity. I wanted to share some ground truths based on personal experience helping family members and friends currently navigating this journey. 📚 Education's True Value The worth of a Masters degree transcends job prospects - it's about deep subject matter expertise and intellectual growth. 🌟 Quality Matters US universities continue to offer world-class research facilities and educational excellence - choose institutions aligned with your specific interests and career goals. 💼 Career Perspective While Big Tech opportunities are attractive, don't make them your sole focus. Many tech giants now have robust operations in India. Evaluate if your goals can be achieved without a Masters in US. 🔍 Beyond Academics Your degree is just one part of the equation. Focus on: Building professional networks Participating in industry events Contributing to research Engaging in hackathons Exploring potential PhD paths 🚀 Pre-Masters Preparation Leverage online resources (MOOCs, YouTube) to build foundational knowledge before your program begins. This gives you a competitive edge for internships and job placements. 💰 Financial Planning Be realistic about the investment. While US opportunities are valuable, maintain a global perspective on career possibilities, including emerging markets. If you're passionate about your field and have done thorough research, don't let market uncertainties deter you. Success comes from strategic planning and staying adaptable. #HigherEducation #MastersDegree #STEM #InternationalEducation #CareerGrowth #USEducation #TechCareers #GlobalOpportunities #StudentLife #CareerAdvice #IndianStudentsAbroad #StudyAbroad #Education2024 #TechEducation #CareerDevelopment

14

Tanay Jaipuria’s analysis on OpenAI’s o-1 model highlights how inference-time scaling laws, allowing models to “think” longer on complex queries, could enhance performance without extra training. This marks a shift toward dynamic reasoning, where compute resources are optimized at inference rather than pre-training. As AI models tackle harder, more consequential tasks, this approach could transform both AI deployment strategies and cost structures. Are inference-time improvements the next key to AI advancement? Let’s discuss!

3

This set of tutorials from Andrej Karpathy, is definitely something I'm looking forward to consuming. It has many lessons on building LLMs from scratch alongwith select sections acceleration, hardware and even multimodal models! Check it out : https://lnkd.in/gEuztn7M #airesearch #largelanguagemodels

23

1 Comment

The next guest speaker in my LLM course is Ram Sriharsha, the CTO of Pinecone, who will talk about Retrieval Augmented Generation (RAG) on Monday October 28. ➡️ Register here: https://lu.ma/r7yokyvc Title: Knowledgeable Models Combine Retrieval with Generation: An Introduction to RAG Date: Monday October 28 from 1:45 PM to 3:15 PM ET Abstract: Large Language Models have emerged into mainstream over the last few years. While they have demonstrated some huge successes already, they have also had challenges with hallucinations, difficulty with dealing with new knowledge and in retrieving relevant, accurate knowledge for downstream tasks. In this talk, we will provide a brief overview of the problem, and discuss a promising approach to solving these problems through what is known as Retrieval Augmented Generation (RAG). Alongside, we will also introduce vector databases, what makes them different from traditional databases and how they fit into RAG. Bio: Dr. Ram Sriharsha is the CTO at Pinecone. At Pinecone, Ram is responsible for overseeing the company’s technical direction in both engineering and research. Prior to joining Pinecone, Ram held numerous senior positions at some of the most well-known companies in the tech industry. Most recently, he was Vice President of Engineering at Splunk. Before that, he was a product and engineering lead at Databricks for their unified analytics platform for genomics. Some years after receiving a PhD in Physics from the University of Maryland, Ram was a Principal Software Engineer and later a Principal Research Scientist at Yahoo. Supplementary Readings: ➡️ The original paper on RAG: "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (https://lnkd.in/e_e56Y54) ➡️ Long context considerations: "Lost in the Middle: How Language Models Use Long Contexts" (https://lnkd.in/ehED53Fe) ➡️ Article on contextual RAG from Anthropic AI (https://lnkd.in/ehK4GwMc)

69

There is a lot of interest in combining Graph Models and Language Models recently. In 2022, I led an initiative at Twitter (now X) in collaboration with Vivek Kulkarni, Ahmed El-Kishky, Sneha Mehta, and executed by our wonderful summer intern Jinning Li called "NTULM: Enriching Social Media Text Representations with Non-Textual Units" on combining large scale graph content at Twitter with Tweet's text content for building better quality Tweet Embeddings. The internal version of the project required addressing quite a few challenges: 1. Finding the node embeddings for NTUs using Twitter Scale Heterogeneous Graph 2. Training BERT with the above scale node embeddings This work was a follow up to my previous collaboration led by Vivek Kulkarni in 2021 on combining social context like time and geography in language models called "LMSOC: An Approach for Socially Sensitive Pretraining". There are a lot of interesting follow ups in this area and combining Graphs with generative LLMs is something which is very effective on real world datasets. NTULM Talk: https://lnkd.in/g2zvmnhh NTULM Paper: https://lnkd.in/gQq6VCQP LMSOC: https://lnkd.in/ghamVwVk

29

1 Comment

Good folks at Meta have introduced Chameleon 🦎 (who names these things? 🤷‍♂️) Chameleon is an AI model that can work with multiple types of data, like text and images, all at once. 🖼️📝 Before you start searching, as of this post, the model/code have not been open-sourced nor is there any commitment to open-source... sorry! 🚫🔓 Still, here is the technical stuff: 👉 Challenges with Current Systems: 📉 Fragmentation: Current multimodal models are often specialized for either text or image tasks, lacking unified approaches. 📊 Scalability: Existing systems struggle with scaling to handle complex, mixed-modal tasks without significant performance degradation. 🔄 Alignment: Aligning textual and visual modalities remains a technical challenge, often requiring separate processing pipelines. 👉 Objective: 🎯 Unified Modeling: Develop a single model capable of handling various multimodal tasks (text generation, image generation, image captioning, visual question answering) seamlessly. 👉 How It's Done 📘 Early-Fusion Architecture 🧠: Utilizes an early-fusion token-based approach to integrate text and image data from the beginning. Stable Training 💪: Implements a tailored alignment recipe and specific architectural parameterization to ensure stability in mixed-modal settings. Broad Evaluation 📊: Assesses the model across various tasks, including visual question answering, image captioning, text generation, image generation, and long-form mixed-modal generation. 👉 Results: (Fun fact they mention Llava-1.5 in comparison but never really share the results) 🏆 Performance: Chameleon achieves state-of-the-art results in image captioning and outperforms models like Llama-2 in text-only tasks. ⚖️ Competitiveness: It shows competitive performance with models such as Mixtral 8x7B and Gemini-Pro. 🖼️ Image Generation: Demonstrates effective non-trivial image generation capabilities within a single model framework. 👩‍⚖️ Human Judgments: Matches or exceeds the performance of larger models, including Gemini Pro and GPT-4V, based on human evaluations for long-form mixed-modal generation. Paper: https://lnkd.in/g4QCi3kG

8

🔮RAG + KG: Powerful Combination for Translation Multi-AI Agents Large language models (LLMs) have shown great promise in machine translation, but they still struggle with contextually dependent terms, such as new or domain-specific words. This leads to inconsistencies and errors that are difficult to address. Existing solutions often depend on manual identification of such terms, which is impractical given the complexity and evolving nature of language. While RetrievalAugmented Generation (RAG) could provide some assistance, its application to translation is limited by issues such as hallucinations from information overload. =================== In this paper, researchers propose CRAT, a novel multi-agent translation framework that: 》 leverages RAG and causality enhanced self-reflection to address these challenges. This framework consists of several specialized agents: ✱ the Unknown Terms Identification agent detects unknown terms within the context, ✱ the Knowledge Graph (KG) Constructor agent extracts relevant internal knowledge about these terms and retrieves bilingual information from external sources, ✱ the Causality enhanced Judge agent validates the accuracy of the information, ✱ Translator agent incorporates the refined information into the final output. =================== This automated process allows for more precise and consistent handling of key terms during translation. Their results show that CRAT significantly improves translation accuracy, particularly in handling context-sensitive terms and emerging vocabulary. --- If your company is looking for an AI consultant to accelerate business growth, or if you are looking to build your career in AI, send me a message and I can help

4

How to understand Gen AI/AI adoption for an organization you are interviewing for ? With so much hype around Gen AI and AI/ML in general, everyone says they use AI/Gen AI due to fomo even if it means they are just using chat gpt from the internet and building some poc's in their local systems. * Remember building pocs is not equal to having production apps used by end consumers. Amongst the orgs I have seen, there are a few categories . Some are research focused(1) , some are research+ implementation focused(2), some are implementation focussed(3) and others just in marketing(4). 4 are the orgs one needs to avoid at all costs. Being involved with orgs in 1 , 2 or 3 depends on a person's personal choice. Try asking the following questions to understand the truth between Gen AI In an org: - What are the No of use cases that are in production and actively used by end customers ? - What are the number of research papers published by the org in Gen AI/AI in the last few quarters.? - What is the estimated business impact of the use cases? - What is the scale and requirment of the use cases. Are we talking about 10 requests a min or 1000 or 10000 req a min.. - Are there specific standardized platforms(cloud/systems/devops/ml/data) and engineering/science practices for doing Gen- AI within the org? - Is llm evaluation, guardrails, monitoring, cost/compute usage and alerting part of the overall existing product? - Are there separate teams or people looking at legal, compliance and security for Gen AI. - Does the org use open source models(mistral, llama)or just enterprise gen ai apis(open ai api's) - Are the orgs fine tuning the existing model with their own data? - If an organisation speaks about rag how many documents are there in their vector store? 10,1000 or 10000 - What is the Gen AI related spending of the organization on research and application building? In the current climate of misinformation and marketing it is important to not be swayed by fancy LinkedIn or media posts but actually go deeper and make career choices. #genai #aiforproduction #llm #mlops #llmops #ai #data

10

Introducing Saaras 1.0, state of the art multilingual speech-to-text translate model, supporting 10 Indian languages. Supports automatic language identification, and contextual biasing using prompts. Sarvam AI

16

The Most Influential AI Week for India - Credit- AIM Amit Raja Naik From high-stakes visits to breakthrough partnerships and bold innovations, this was no ordinary week for AI. It was a game-changer, cementing India's place in AI history.  Here are the key highlights from NVIDIA AI Summit India 2024:  1- NVIDIA and Reliance Partnership for AI Infrastructure: Huang announced a landmark partnership with Reliance Industries, aiming to scale AI infrastructure with the latter’s $10 billion investment in data centres across India. Reliance is building infrastructure for 1 GW at Jamnagar, which is scalable for multi-GW use, backed by green energy, to support NVIDIA’s latest GPUs and AI models. 2-20x Increase in AI Compute Power in India by Year-End: NVIDIA looks to multiply its compute infrastructure in India by 20 times over the next year, supported by partners like Yotta, Tata Communications, and E2E, positioning India as a global leader in generative AI. 3-Development of LLMs in Indian Languages: Acknowledging India's diverse linguistic landscape, NVIDIA is collaborating to create a Hindi large language model and potentially other regional models to bridge the digital divide across India’s 25 formal languages and 1,500 dialects. Jensen noted that “this is the hardest language model region in the world”, underscoring the unique challenges and strategic advantage in building AI for a multilingual society. 4-AI’s Exponential Compute Demand and Blackwell GPU Launch: Jensen introduced NVIDIA’s Blackwell GPUs, designed to handle the 4x increase in compute demand needed each year for training ever-larger models. Blackwell racks, capable of delivering 120 kW of compute power, can train models with over a trillion parameters, with shipment to India scheduled for Q4 2024. 

14

Text chunks in retrieval may risk losing context. The below article introduces contextualizing each chunk using an LLM. The added cost and latency required for contextualizing is partially addressed through a lightweight LLM (haiku) and prompt caching. Along with other known techniques (BM25, reranking, etc.), a significant reduction in retrieval failure is achieved. Overall, a good article to read. https://lnkd.in/g_5fgAN8

10

Going from PoC to Production for LLMs? 🤔 In the age of GPTs, chat interfaces are ubiquitous. But what if we add a voice/conversational layer on top? Imagine a C-suite member checking sales on the go or a new employee understanding leave policies through a natural voice interface. Intriguing, right? This got me thinking: how do we move from a PoC running on a local machine to a full-fledged production application? Here's what I found out: 1️⃣ Data Management and Knowledge Distillation: Handle and version your knowledge base (PDFs, Images, Excel, ER Diagrams). Keep data up-to-date and manage new information efficiently. How do you ensure information in the data is not outdated? What’s the best way to encode and chunk this data? The questions arise! 2️⃣ Access control and Citations: How do you ensure people cannot access data they are not supposed to have access to? How do you retrieve specific documents to ensure better answers (LLM models work the best when provided with specific information) and lower bills (Each LLM service charges by the number of tokens)? How do you remove any Personally Identifiable Information (PII)? What kind of caching strategies to use? (Pareto principle, 80% of user queries would be for 20% of the questions that get repeated) 3️⃣ Security and Reproducibility: Protect against bad actors and prompt injection. Create secure networks. How do you safeguard your application? It’s a growing field with evolving challenges. 4️⃣ Heart of the Operations (Sort of, I still think your knowledge base is the heart): Now that you have designed and taken care of the above aspects, there is still more. What size of the model to use? What environment to host it in? how to scale and replicate? (If you have a fine-tuned model). 5️⃣ Generating responses: How do you ensure the results generated are accurate and correct? While it is an active area of research, the only effective solution at this point is humans in the loop. Productionizing LLMs is complex, with more questions than answers. I’d love to hear your experiences and insights. What challenges have you faced, and what solutions have you found? If you can think of important things that I may have missed out, feel free to comment below! Ps. I will be writing and building more on this idea in the coming days, stay tuned for more :) #LLM #AI #MachineLearning #DataScience #VoiceInterface

12

1 Comment

📊 The Power of High-Quality, Diverse, Real-World Data Fine-Tuning LLMs 🌍 When it comes to training Large Language Models (LLMs), the mantra isn’t just “more data”—it’s better data. While having more data can be beneficial, it’s the quality, diversity, and real-world relevance of the data that truly drive impactful results. Here’s why: 1️⃣ Higher Quality: clean, well-curated data improves model accuracy and reduces noise, making the model more reliable and stable. Poor-quality data often leads to inconsistent responses and increased hallucinations. Unbiased labeling consistency is key. 2️⃣ Diversity: it ensures the model can generalize across different contexts and user scenarios. This makes the model more robust, safe and inclusive, which is especially important in NLP for avoiding bias and ensuring representation. 3️⃣ Real-World Datasets: training on real datasets helps models perform better in real-world applications, as it learns from actual patterns and usage scenarios, making the model more practical and effective. For LLMs, there’s a general playbook to prepare data effectively: (1) Collect Instruction-Response Pairs: gather examples of how tasks should be answered, covering various domains and complexity levels. (2) Concatenate Pairs: organize the data into structured sequences, allowing the model to learn from end-to-end interactions. (3) Tokenize: process the data into tokens, using padding and/ or truncation as needed to maintain uniform length. I particularly love 🤗 transformers library's AutoTokenizer to identify which tokenizer to use on every model. (4) Split into Train/Dev/Test Sets: partition your data into training, validation, and testing sets to ensure proper model evaluation and tuning. Ultimately, data quality prevails over quantity, specially as we live in the LLMs era when models have been pre-trained with almost all the public data available in the 🌎 . A well-prepared dataset tailored to your application will make all the difference in the performance and reliability of your model. #AI #MachineLearning #DataQuality #LLM #NLP #ModelTraining #DataPreparation

5

Interesting to see how the Bhagavad Gita is being reimagined through AI. Great to see how Replit is making building accessible - from idea to implementation in a weekend.

8