Ajay Halthor

🚀 Making LLMs work for you - A Guide to Become 10x Programmer/MLE/DS I am sure many of you use LLMs for your day-to-day work. Mostly, I have seen people use LLMs while writing code (Copilot), or as an alternative to Copilot to catch bugs, but they have become much more useful than that, provided you know how to use them well. Here are a few ways I have been using LLMs in my daily work that are helping me increase my productivity: 📚 1. Quick Paper Assessment with ELI5 Summaries - Paper abstracts are called abstracts for a reason. Asking an LLM for an ELI5 summary makes it much easier to know if I want to read a paper. 💻 2. Implementing Research Papers in Code - Try giving an LLM like Claude/ChatGPT a research paper and ask to code it. You would be surprised. Tip: Claude performs much better for this task. 🔄 3. Upgrading Code Structure and Readability - Do you want to add typing hints when writing your Python code, or do you need comments in your code, or do you have a big function that you want to be converted to classes in your code? LLMs can do it all, and you would be amazed at the code-conversion power they have. ⚡ 4. Optimizing Code Performance - The thing with LLMs is that if you tell them what to do, they can do it well. For example, if you give them a non-optimized piece of code that runs correctly, they will be able to write it in a much-optimized way. For example, I just used an LLM to convert a matrix multiplication in PyTorch to sparse MM as I was getting an OOM memory error because one of the matrices became too large. I am not even joking, but that would have taken at least 1-2 hours for me to come up with. 📝 5. Automated Documentation - Got bored of writing documentation for what you did but have the code? LLMs also work the other way around. Want to write a summary of what you did in a week, you can use an LLM for that. Saves so much time that gets wasted in formatting. 🧪 6. Streamlined Test Framework Creation - Let LLMs handle the boilerplate for Unit tests & Integration tests while you focus on core logic. 🌐 7. Language-Agnostic Development - You can paste a function in Python and ask for its equivalent in Rust, complete with language-specific best practices and error handling. This dramatically reduces the learning curve when working with new languages and helps maintain productivity during language transitions. 🤝 Apart from all of the above, LLMs work well as mentors as well as interns. LLMs are great when you just need someone to bounce ideas with. I sometimes just talk to an LLM as I would to a colleague talking about the problems I have and how I can solve them. And they come up with the right ideas or send me in the right direction a lot of times. They also act as a personal assistant for free, who can deal with writing messages, creating charts, and doing all sorts of uninteresting work for you. 💭 What's your favorite way to use LLMs? Let me know in the comments! #MachineLearning #AI #Programming

47

Tanay Jaipuria’s analysis on OpenAI’s o-1 model highlights how inference-time scaling laws, allowing models to “think” longer on complex queries, could enhance performance without extra training. This marks a shift toward dynamic reasoning, where compute resources are optimized at inference rather than pre-training. As AI models tackle harder, more consequential tasks, this approach could transform both AI deployment strategies and cost structures. Are inference-time improvements the next key to AI advancement? Let’s discuss!

3

Good folks from Carnegie Mellon University's WAVLab have released XEUS - A Cross-lingual Encoder for Universal Speech 🌐🗣️ Simple speech representation is hindered by a lack of labelled data. That's where Self-supervised learning (SSL) comes in! 🔍📚 XEUS is a SoTA speech encoder trained in 4000+ languages! 🏆🌍 Before this, I did not even know that there are 4000+ languages. But the paper says this is a small dent in the world's 7000+ languages. 🌐🗺️ More details: - 577M parameters 📊 - 1M+ hours of speech training data across 4057 languages 🗣️⏳ - Uses the E-Branchformer architecture and is trained using HuBERT-style masked prediction of discrete speech tokens 🏗️🔠 - For robustness, the input speech is also augmented with acoustic noise and reverberation 🎙️🔊 Beats both MMS 1B and w2v-BERT v2 2.0 🥇🏆 It requires fine-tuning to be used in downstream tasks such as Speech Recognition or Translation. 🛠️🌐 Its hidden states can also be used with k-means for semantic Speech Tokenization. 🗣️🔑 Release: https://lnkd.in/gqrfvsRF

11

There is a lot of interest in combining Graph Models and Language Models recently. In 2022, I led an initiative at Twitter (now X) in collaboration with Vivek Kulkarni, Ahmed El-Kishky, Sneha Mehta, and executed by our wonderful summer intern Jinning Li called "NTULM: Enriching Social Media Text Representations with Non-Textual Units" on combining large scale graph content at Twitter with Tweet's text content for building better quality Tweet Embeddings. The internal version of the project required addressing quite a few challenges: 1. Finding the node embeddings for NTUs using Twitter Scale Heterogeneous Graph 2. Training BERT with the above scale node embeddings This work was a follow up to my previous collaboration led by Vivek Kulkarni in 2021 on combining social context like time and geography in language models called "LMSOC: An Approach for Socially Sensitive Pretraining". There are a lot of interesting follow ups in this area and combining Graphs with generative LLMs is something which is very effective on real world datasets. NTULM Talk: https://lnkd.in/g2zvmnhh NTULM Paper: https://lnkd.in/gQq6VCQP LMSOC: https://lnkd.in/ghamVwVk

29

1 Comment

Meta’s new language model "Llama 3" is finally here. Plus, you can access it directly with chatGPT like interface. (not available in India yet) ► Integration with Meta AI: Llama 3 is now part of Meta AI, facilitating an array of tasks including coding and problem-solving. ► Model Variants: Available in both 8B and 70B configurations, offering versatility for different development needs. ► Enhanced Training: Trained on a 15T token dataset, 7x larger than Llama 2, with four times more code content. ► Extended Context Length: Supports an 8K context length, doubling the capacity of the previous model, Llama 2. ► Infrastructure: Utilizes Meta's recently built 24K GPU clusters, enhancing its data handling and scalability capabilities. ► Access and Usage: Users can interact with Llama 3 through a ChatGPT-like interface, enhancing user accessibility and ease of use.

4

How are Graph Convolutional Neural Networks (GCNNs) a generalisation of Convolutional Neural Networks (CNNs)? My next blog in the Graph series on "Deep Learning with Graphs" is on the way (well, it's already written on paper) and it's going to be a blockbuster. The beauty of Graph Machine Learning is that the more you read about it, the more you feel that your neurons (real ones!) are learning to think better! And you're already in love with it! Coming to the point: One might recall the convolution operations in a CNN. You have a grid of pixels representing an image. You have a 3 x 3 filter. You slide the filter over the image and carry out aggregation to obtain a new pixel value. You proceed in the same fashion over the whole grid, sliding towards the right , bottom, carrying out the convolution operation and finally getting a new transformed, reduced image through the series of aggregations. You repeat this through the different convolutional layers which together form a Convolutional Neural Network Now think of a Graph. It is not simple grid like an image. The topology of a Graph is arbitrary. There is no notion of left, bottom, right in a Graph like in an image which is represented through a grid of pixels. You cannot slide over a Graph as simplistically you did and performed the convolutional operations in an image. How do you approach the problem? How do you carry out convolutional operations in a Graph? Now take a node in the Graph. Take all the nodes which are the immediate neighbours of this node. The convolutional operation here is: You aggregate the information from all the neighbours and pass it to the node. The information is nothing but the emedfings of the neighbour nodes. You add together the emedding of the node under consideration. You transform the aggregated embedding by multiplying with a weight Matrix and transform the node's own embedding by multiplying with a different weight matrix. This is the typical linear transformation operation in a NN. You apply non linearity - the activation function like ReLu. Now think about a CNN taking the same grid and considering a central pixel. Consider the rest of the pixels around the central pixel. You are essentially doing the same thing: taking information from the neighbour pixels along with the central pixel, aggregating, as I described above for the node in the Graph. Hence a Graph Convolutional Network is a generalisation of CNN The story goes, you do the same for every node on the Graph. Each node in the Graph represented through a different neural network architecture! This is amazing 🔥🔥🔥💓💓💓 #graphs #machinelearning #deeplearning My Technical Blogs Library: 1) https://lnkd.in/ejyWacXn 2) https://lnkd.in/bfrkRjq Follow Ajay Taneja for insights on: Data Science, Data Engineering, Finite Element Analysis, Non Linear Finite Element Analysis, Fatigue, Fracture Mechanics

22

Good folks at Meta has just released Llama 3.2, their latest advancement in large language models. Here are the key highlights: 🔹 First-ever multimodal Llama models: Llama 3.2 11B Vision and 90B Vision can process both text and images, opening up new possibilities for visual reasoning and image-based tasks. (They are not diffusion models) 🔹 Lightweight models for edge devices: Llama 3.2 1B and 3B are optimized for mobile and edge computing, enabling on-device AI applications with reduced latency. 🔹 Impressive context length: All models support up to 128,000 tokens, allowing for processing of much longer inputs. 🔹 Advanced architecture: Llama 3.2 uses an optimized transformer architecture with grouped query attention (GQA) for improved efficiency. 🔹 Extensive training: The vision models were trained on 6 billion (image, text) pairs, with a knowledge cutoff of December 2023. 🔹 Multilingual support: While primarily optimized for English, the text models also support German, French, Italian, Portuguese, Hindi, Spanish, and Thai. 🔹 Competitive performance: Evaluations show Llama 3.2 models performing on par with leading foundation models like Claude 3 Haiku and GPT4-mini on various benchmarks. 🔹 Responsible AI focus: The models incorporate safety mitigations and have undergone supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF). 🔹 Open-source availability: Llama 3.2 is being released as an open-source model, fostering innovation and collaboration in the AI community. This release marks a significant step forward in Meta's AI capabilities, potentially rivaling offerings from OpenAI and Anthropic. Developers and researchers, are you excited to explore Llama 3.2's potential? Let's discuss in the comments!

13

📌LLaMA-Omni: Seamless Speech Interaction with Large Language Models Models like GPT-4o enable real-time interaction with large language models (LLMs) through speech, significantly enhancing user experience compared to traditional text-based interaction. However, there is still a lack of exploration on how to build speech interaction models based on open-source LLMs. 🔗https://lnkd.in/dchmdne2 #LargeLanguageModels

1

🎓 Is a Masters in USA in Computer Sciene/Data Science still worth it? There is a lot of speculation around this question, with clickbait news coming up with random facts and suggestions. The economy and job market has been tough in US and across the world in 2024 and there is uncertainties around visa and work permit. Lots of student are yet to land an internship or job offer with surmounting pressure of visa validity. I wanted to share some ground truths based on personal experience helping family members and friends currently navigating this journey. 📚 Education's True Value The worth of a Masters degree transcends job prospects - it's about deep subject matter expertise and intellectual growth. 🌟 Quality Matters US universities continue to offer world-class research facilities and educational excellence - choose institutions aligned with your specific interests and career goals. 💼 Career Perspective While Big Tech opportunities are attractive, don't make them your sole focus. Many tech giants now have robust operations in India. Evaluate if your goals can be achieved without a Masters in US. 🔍 Beyond Academics Your degree is just one part of the equation. Focus on: Building professional networks Participating in industry events Contributing to research Engaging in hackathons Exploring potential PhD paths 🚀 Pre-Masters Preparation Leverage online resources (MOOCs, YouTube) to build foundational knowledge before your program begins. This gives you a competitive edge for internships and job placements. 💰 Financial Planning Be realistic about the investment. While US opportunities are valuable, maintain a global perspective on career possibilities, including emerging markets. If you're passionate about your field and have done thorough research, don't let market uncertainties deter you. Success comes from strategic planning and staying adaptable. #HigherEducation #MastersDegree #STEM #InternationalEducation #CareerGrowth #USEducation #TechCareers #GlobalOpportunities #StudentLife #CareerAdvice #IndianStudentsAbroad #StudyAbroad #Education2024 #TechEducation #CareerDevelopment

14

Back in 2017, the game-changing paper “Attention Is All You Need” by Ashish Vaswani and his team pretty much set up today’s Large Language Models (LLMs) like GPT and BERT for success. Instead of forcing models to remember everything in sequence (like a talkative friend who won’t skip a single detail), this “attention” mechanism lets them focus only on the important stuff—faster, smarter, and with a bit more personality. It’s the secret sauce behind LLMs. Curious to dig deeper? Check out the original paper that started it all: https://lnkd.in/gAbU_urJ #LLM #AITrends #GPT #transformers #AIResearch

10

This set of tutorials from Andrej Karpathy, is definitely something I'm looking forward to consuming. It has many lessons on building LLMs from scratch alongwith select sections acceleration, hardware and even multimodal models! Check it out : https://lnkd.in/gEuztn7M #airesearch #largelanguagemodels

23

1 Comment

Seeing companies in the AI space as early as seed stage compete with the likes of DeepMind, OpenAI, & Meta on compensation packages – with some surpassing $1M in cash offers   A common theme? Researchers leaving big tech and raising a lot of VC money   Ilya Sutskever leaves OpenAI and starts Safe Superintelligence, raises $1BN Series-A round Anthropic was born after an internal conflict at OpenAI surrounding commercialisation, they’ve raised close to $10BN in 3 years The three co-founders of Mistral AI, a Series-B AI company that has raised €1.1BN to-date are all ex-DeepMind and Meta AI researchers Inflection AI was founded by a DeepMind co-founder & one of their researchers, and havr raised $1.5BN Adept have raised $415M and was founded by former Google & OpenAI researchers Now there is a lot more competition out there in what is a very immature talent market Investment levels are bound to drop, there’s a lot of hype around AI right now – although I expect it will survive the bounce a lot better than the crypto space did Are these comp levels sustainable, or is it a race to the top? The next 2 years will be interesting

33

1 Comment

Almost 67% of organizations use GenAI products that rely on LLMs to work with human language and produce content. And by 2025, 50% of digital work is estimated to be automated through apps using LLMs 🤯 Generative AI boom is real! So is to up skill yourself in it! So how do you go about it? I'd recommend you to read books and enrol in courses developed by Industry experts. I am thankful to Packt for sending this amazing book to me ❤️ A great curation by Paul Singh and Anurag(Anu) Karuparti! This book helps you: ✅ Gain foundational understanding of GenAI and LLMs ✅ Learn how NLP has evolved over the years ✅ Understand how to fine tune LLMs and build RAG apps ✅ Go through Prompt Engineering best practices ✅ Grasp the importance of Responsible AI and how to scale - secure GenAI systems An all in one book for beginners so say the least! I liked it and I highly recommend you to read it. Up skill yourself because it helps you get the competitive edge and keeps you on a safer side in an ever changing job market. Follow Bhavishya Pandit for more posts on Gen AI/ LLMs/DS #datascience #ai #ml #python #nlp #genai #generativeai #microsoft #packt #openai #gpt4 #machinelearning #naturallanguageprocessing #ibm #artificialintelligence

172

18 Comments

🔮RAG + KG: Powerful Combination for Translation Multi-AI Agents Large language models (LLMs) have shown great promise in machine translation, but they still struggle with contextually dependent terms, such as new or domain-specific words. This leads to inconsistencies and errors that are difficult to address. Existing solutions often depend on manual identification of such terms, which is impractical given the complexity and evolving nature of language. While RetrievalAugmented Generation (RAG) could provide some assistance, its application to translation is limited by issues such as hallucinations from information overload. =================== In this paper, researchers propose CRAT, a novel multi-agent translation framework that: 》 leverages RAG and causality enhanced self-reflection to address these challenges. This framework consists of several specialized agents: ✱ the Unknown Terms Identification agent detects unknown terms within the context, ✱ the Knowledge Graph (KG) Constructor agent extracts relevant internal knowledge about these terms and retrieves bilingual information from external sources, ✱ the Causality enhanced Judge agent validates the accuracy of the information, ✱ Translator agent incorporates the refined information into the final output. =================== This automated process allows for more precise and consistent handling of key terms during translation. Their results show that CRAT significantly improves translation accuracy, particularly in handling context-sensitive terms and emerging vocabulary. --- If your company is looking for an AI consultant to accelerate business growth, or if you are looking to build your career in AI, send me a message and I can help

4

The whole field of prompt engineering can be reduced to these 6 techniques I use almost daily when using ChatGPT (or other LLMs). Here they are ↓ #1. Few shot prompting Add to your prompt 2 or 3 high-quality demonstrations on the target task, each consisting of both input and desired output. The LLM will better understand your intention and what kind of answers you expect based on concrete examples. #2. Self-consistency sampling Sample multiple outputs with "temperature > 0" and select the best one out of these candidates. How to pick the best candidate? It will vary from task to task, but here are 2 primary scenarios ↓ 1. Some tasks are easy to validate, such as programming questions. In this case, you can write unit tests to verify the correctness of the generated code. 2. For more complicated tasks, you can manually inspect them or use another LLM (or another specialized model) to rank them. #3. Chain-of-Thought (CoT) You want to force the LLM to explain its thought process, which eventually leads to the final answer, step by step. This will help the LLM to reason complex tasks better. You want to use CoT for complicated reasoning tasks + large models (e.g., with more than 50B parameters). Simple tasks only benefit slightly from CoT prompting. Here are a few methods to achieve CoT: - provide a list of bullet points with all the steps you expect the LLM to take - use "Few shot prompt" to teach the LLM to think in steps ... or my favorite: use sentences such as "Let's think step by step." #4. Augmented Prompts The LLM's internal knowledge is limited to the data it was trained on. Also, often, it forgets specific details of older training datasets. The most common use case is Retrieval-Augmented Generation (RAG). That is why using the LLM as a reasoning engine is beneficial to parse and extract information from a reliable source of information given as context in the prompt. 𝘞𝘩𝘺? - avoid retraining the model on new data - avoid hallucinating - access to references on the source #5. A single responsibility per prompt Quite self-explanatory. It is similar to the DRY principle in SWE. Having only x1 task/prompt is good practice to avoid confusing the LLM. If you have more complex tasks, split them into granular ones and merge the results later in a different prompt. #6. Be as explicit as possible The LLM cannot read your mind. To maximize the probability of getting precisely what you want, you can imagine the LLM as a 7-year-old to whom you must explain everything step-by-step to be sure he understood. . The truth is that prompt engineering is quite intuitive, and we don't have to overthink it too much. What would you add to this list? #machinelearning #mlops #datascience

194

12 Comments

I wanted to share an update with you. Two weeks ago, I left Amazon after a four-year-long stint. Tomorrow, I will join Divyam.AI – an early-stage start-up. Lastly, I continue to be a doctoral student at IIT, Kharagpur. Here's a somewhat personal reflection:- Amazon: – We studied the principles underpinning two-sided marketplaces for incentives (e.g., funders and customers being the two sides, with coupon as the incentive), and documented the findings in 3 internal and 1 external peer-reviewed publications. – We built two 0-1 products for optimal incentivization: one deployed at Amazon Pay, and the other gaining rapid adoption amongst amazon.in categories. – My greatest satisfaction came from an unexpected place – auto-rickshaws in Bangalore. They sport Amazon Pay stickers with the tagline "Earn exciting rewards"! – The greatest gift from Amazon was a rather powerful realization: "In the midst of winter, I found there was, within me, an invincible summer" (Albert Camus). Divyam.AI: – "The future is already here – it's just not evenly distributed" (William Gibson): I believe LLM is the future in many walks of life, but its "tokenomics" should align with the interests of its operator, not the infrastructure provider. – We are building a self-driving LLM inference stack that exploits full-stack (Systems + ML) optimization opportunities in order to deliver an unprecedented cost-performance Pareto to its operator. IIT, Kharagpur: – I managed to complete the coursework, and learnt new principles/tools, e.g., measure-theoretic probability. I also managed to clear the Comprehensive Examination – a close shave, though ;) – I continue to study how human behaviour is re-shaped under nudges and incentives, with Prof. Niloy Ganguly, my supervisor.  – I am equally excited about this pursuit, where I only strive to understand the gestalt – not to profit from it.

525

83 Comments