Akshay Trivedi

🚀 Making LLMs work for you - A Guide to Become 10x Programmer/MLE/DS I am sure many of you use LLMs for your day-to-day work. Mostly, I have seen people use LLMs while writing code (Copilot), or as an alternative to Copilot to catch bugs, but they have become much more useful than that, provided you know how to use them well. Here are a few ways I have been using LLMs in my daily work that are helping me increase my productivity: 📚 1. Quick Paper Assessment with ELI5 Summaries - Paper abstracts are called abstracts for a reason. Asking an LLM for an ELI5 summary makes it much easier to know if I want to read a paper. 💻 2. Implementing Research Papers in Code - Try giving an LLM like Claude/ChatGPT a research paper and ask to code it. You would be surprised. Tip: Claude performs much better for this task. 🔄 3. Upgrading Code Structure and Readability - Do you want to add typing hints when writing your Python code, or do you need comments in your code, or do you have a big function that you want to be converted to classes in your code? LLMs can do it all, and you would be amazed at the code-conversion power they have. ⚡ 4. Optimizing Code Performance - The thing with LLMs is that if you tell them what to do, they can do it well. For example, if you give them a non-optimized piece of code that runs correctly, they will be able to write it in a much-optimized way. For example, I just used an LLM to convert a matrix multiplication in PyTorch to sparse MM as I was getting an OOM memory error because one of the matrices became too large. I am not even joking, but that would have taken at least 1-2 hours for me to come up with. 📝 5. Automated Documentation - Got bored of writing documentation for what you did but have the code? LLMs also work the other way around. Want to write a summary of what you did in a week, you can use an LLM for that. Saves so much time that gets wasted in formatting. 🧪 6. Streamlined Test Framework Creation - Let LLMs handle the boilerplate for Unit tests & Integration tests while you focus on core logic. 🌐 7. Language-Agnostic Development - You can paste a function in Python and ask for its equivalent in Rust, complete with language-specific best practices and error handling. This dramatically reduces the learning curve when working with new languages and helps maintain productivity during language transitions. 🤝 Apart from all of the above, LLMs work well as mentors as well as interns. LLMs are great when you just need someone to bounce ideas with. I sometimes just talk to an LLM as I would to a colleague talking about the problems I have and how I can solve them. And they come up with the right ideas or send me in the right direction a lot of times. They also act as a personal assistant for free, who can deal with writing messages, creating charts, and doing all sorts of uninteresting work for you. 💭 What's your favorite way to use LLMs? Let me know in the comments! #MachineLearning #AI #Programming

47

What’s the Real Context Size of Your Long-Context LLMs? Picking up from my last post (https://lnkd.in/gYtgs_DT), where I talked about the pros and cons of long-context LLMs in Retrieval-Augmented Generation. This NVIDIA research takes a closer look at how these models perform at their limits, with insightful findings that reveal some non-obvious challenges. The authors tested 17 language models, including well-known ones like Gemini 1.5, GPT-4, and Llama 3.1. Turns out, despite many models claim to handle huge contexts, very few actually keep up their performance as context length grows. This might come as a surprise if you're familiar with traditional benchmarks like "needle-in-a-haystack" and passkey retrieval, where most models achieve near-perfect accuracy. While those tests are useful, the new RULER benchmark pushes LLMs with more complex tasks, such as multi-hop tracing and aggregating information across long contexts, ultimately revealing their true limitations and areas for improvement. As language models grow in size and context capacity, it's easy to get excited about their new capabilities and forget the trade-offs. Just because a model can handle massive contexts doesn’t mean more is always better — often, keeping it small and simple still wins the day. I'll drop the link to the paper in the comments. #MachineLearning #ArtificialIntelligence #AI #LLM #GenAI #NeuralNetworks

57

3 Comments

𝗟𝗹𝗮𝗺𝗮 𝟯 𝘂𝗻𝘃𝗲𝗶𝗹 𝗮 𝗳𝘂𝗻𝗱𝗮𝗺𝗲𝗻𝘁𝗮𝗹 𝗰𝗼𝗺𝗽𝘂𝘁𝗶𝗻𝗴 𝘁𝗿𝗮𝗱𝗲𝗼𝗳𝗳 𝘁𝗵𝗮𝘁 𝗳𝗲𝘄 𝗮𝗿𝗲 𝘀𝗲𝗲𝗶𝗻𝗴⬇️ 🦙Llama 3 just came out and to the surprise of many people, the L3-8B version can match the L2-70B one on several benchmarks. Several people also argued that the chinchilla paper (https://lnkd.in/dXaRqyea) was wrong because the llama 3 scaling is very far from chinchilla-optimal. 🤔 In my opinion, this is due to a misunderstanding of what the chinchilla paper is about and more generally of the effects of scale in deep learning. The bitter lesson (https://lnkd.in/d3sVYYgp) has become common knowledge in the field thanks to the GPT family that popularized it. We hear everywhere that "bigger models are better": Yes it's true, but in what sense? Many people have a mental model along the lines of "bigger models are more capable" meaning they would be able to achieve levels of performance that smaller models can't. ❌ This is not true with LLMs and is mainly because we train only once on tokens (most of them at least) and therefore overfitting is not possible. The 𝘳𝘦𝘢𝘭 advantage of bigger models over smaller ones is that they converge *faster*. In other words, you can reach a given level of NLL (modeling performance) with fewer training iterations. ❔ Then a natural question arises: Should I train a smaller model with more or a bigger model with fewer iterations? That's where the chinchilla paper brings a response. They say that the question they are answering is: "Given a fixed FLOPs budget, how should one trade-off model size and the number of training tokens?" (implying "to achieve the best possible NLL"). 🧐 Coming back to Llama 3, it's obvious that the Llama team is aware of this work and there is a reason why they didn't follow this recipe. My guess is that they are taking into account the inference cost in the balance (which is completely out of the scope of the chinchilla paper). When one "overtrains" an 8B model, one ends up with a worse "NLL to training compute" ratio, but also with a better "NLL to inference compute" ratio. ✅ Now if we think about it from a business perspective, it's a very powerful tradeoff because you can choose to buy performance either at a fixed cost (training) or at a variable one (inference). Hope this post helped you! #Llama3 #Meta #LLM #AI

9

3 Comments

Optillm: An optimizing inference proxy for LLMs Optillm is an optimizing inference proxy that has over a dozen techniques that aim to improve the accuracy of the responses using test-time compute. Over the last couple of months we have set several SOTA results using smaller and less capable models like gpt-4o-mini. Recently, we have added support for plugins that enable capabilities like memory, privacy and code execution to optillm. Plugins are just python scripts that you can also write yourself, optillm would then load them at start from the directory. You can now also combine the plugins and techniques using & and | operators. E.g. We recently evaluated the new FRAMES benchmark from Google. Using a combination of plugins and techniques (we used readurls&memory-gpt-4o-mini) we were able to get 65.7% accuracy on the benchmark which is very close to what Google reported in their paper with Gemini Flash 1.5 (66.5) which has a context length that is almost 10 times that of gpt-4o-mini. Do check out Optillm at https://lnkd.in/gN6_kNky

21

1 Comment

Prof. Rao's post is a good starting point for LLM practitioners to become familiar with classic (i.e., predating LLMs) AI concepts and theories, for two reasons: 📌 To understand that some things are theoretically beyond the reach of LLMs and not waste resources chasing them. 📌 To hedge against the possibility that LLMs may be replaced by other technologies and/or undergo fundamental technological revamps, which would render all existing investments in LLMs obsolete. #artificialintelligence #machinelearning #deeplearning

41

3 Comments

I remember when this all started. Faris - invited me to catch up at our favorite spot around Blok M. I had no idea he’d just wrapped up a procedure at Pondok Indah Hospital! (He really knows how to add drama to our catch-ups.) We’d been on separate paths and projects since "the winding down project", but there he was, reminding me of our dream to ride this AI wave together again. It was a plan we’d hatched with Sahal Zain & Gurnoor Dhillon when the world of AI was still mostly theoretical and jargon-heavy. Somehow, the conversation continued with Gurnoor Dhillon while he was waiting for his meeting with someone who would eventually become one of CalvinBall’s key early adopters. It felt like everything was coming together, like a series of seemingly random dots connecting to form a bigger picture. Seeing how far we’ve come as a team at CalvinBall Technologies is truly exciting. What started as theoretical discussions has now evolved into practical Gen AI solutions that are solving everyday problems and hold endless possibilities to shape the industry. 👨‍🚀 👩‍🚀 Here’s to pushing boundaries and achieving new milestones together! Gurnoor Dhillon, Sandeep Ramesh, Faris -, Sahal Zain, Eva Zuliya, Ratu Nadiah Khairunnisa 🚀 #GenAI #AIInnovation #TechJourney #ArtificialIntelligence #TechTransformation #CalvinballTech #Calvinballers

55

2 Comments

Chip Huyen on Twitter today below. I 100% agree. Metric hygiene is system hygiene. LLM-Generated metrics require a higher standard of care than they're currently receiving, and much more scrutiny. Mind the compasses. 🐦  A big issue I see with AI systems is that people aren't spending enough time evaluating their evaluation pipeline. 1. Most teams use more than one metrics (3-7 metrics in general) to evaluate their applications, which is a good practice. However, very few are measuring the correlation between these metrics.If two metrics are perfectly correlated, you probably don't need both of them. If two metrics strongly disagree with each other, either this reveals something important about your system, or your metrics just aren't trustworthy.2. Many (I estimate 60 - 70%?) use AI to evaluate AI responses, with common criteria being conciseness, relevance, coherence, faithfulness, etc. I find AI-as-a-judge very promising, and expect to see more of this approach in the future.AI-as-a-judge scores aren’t deterministic the way classification F1 scores or accuracy are. They depend on the judge's model, the judge's prompt, and the use case. Many AI judges are good, but many are bad.Yet, very few are doing experiments to evaluate their AI judges. Are good responses given better scores? How reproducible the scores are -- if you ask the judge twice, do you get the same score? Is the judge's prompt optimal? Some aren’t even aware of the prompts their applications are using, because they use prompts created by eval tools or by other teams.Also fun fact I learned from a (small) poll yesterday: some teams are spending more money on evaluating responses than on generating responses 🤯 https://lnkd.in/gYnTRZW3

2

November 2022 – The “Big Bang” of the ML/AI vs. SWE Hiring Divide ML/AI Engineers and Software Engineers used to largely look and smell the same. Similar comp. Similar hiring trends. And for most companies, ML/AI engineers and SWEs were historically lumped into the same core SWE job family. That is, until November of 2022. __________________ November 2022: “The Big Bang” November 30, 2022. That’s when ChatGPT launched and took the world by storm. Remember the Christmas '22 magic of sharing GPT with your elderly relatives and watching their eyes bulge? But it wasn’t just GPT that took the world by storm. Also in November of 2022, the labor market experienced a dramatic shift specific to ML/AI talent, fueled directly by the AI wave/hype cycle. __________________ A few weeks ago, I posted about the emerging compensation premium for ML/AI engineers over SWE Generalists. TLDR => there’s currently a ~8-25% premium for the majority of ML/AI IC Engineers over SWE Generalists at public tech companies, with an increasing premium for more senior levels. So, why and how is the “ML/AI Premium” emerging? __________________ Attached is a chart showing the number of ML/AI engineers vs. SWE Generalists hired in Pave’s over the past ~four years, normalized by the number of new hires in Q1 2020. There is a clear divide that is introduced in Q4 2022. Since that point in time, the number of ML/AI Engineers being hired has skyrocketed versus SWE Generalists. __________________ Two theories for why: 1️⃣ Simply, the demand for AI/ML talent jumped overnight. AI/ML engineers who were previously content with their current jobs suddenly started getting poached with increasing aggression. 2️⃣ From a job architecture perspective, I speculate that more companies began making the job family distinction between ML/AI Engineers and SWE Generalists around this point in time to account for the increasingly complex and rapidly evolving compensation treatment needed for the ML/AI engineers. __________________ One caveat: the "Deep AI Researcher” vs. the “Applied ML/AI Engineer” I talked with a member of OpenAI’s technical staff yesterday who made a key point to me. In short, perhaps "~1%" of “AI Engineers” are the deep researchers actively creating the new LLM and GenAI architectures evolving before our eyes. And this “1%” is getting compensated orders of magnitude more than anyone else. Meanwhile, the other "~99%" of “AI Engineers” are the applied ML/AI engineers adopting the latest models off the shelves. And it is this cohort that is being comp’d well...but not orders of magnitude higher than requisite SWE peers. Pave’s Machine Learning job family largely corresponds with the 99% rather than the 1%. The 1%, meanwhile, is tough to track, but we are working on it. Good luck out there. #pave #marketdata #machinelearning #artificialintelligence #trends

87

17 Comments

Facebook just released a very different kind of LLMs. These are focused on code optimization and compiler optimization. This is super important because it can fundamentally change the way code is executed on any machine. Here's how it works right now: Programs written in many languages like C/C++ have intermediate steps before they are run on the computer. For example they are converted into Byte Code and then converted into machine code via LLVM (Low level virtual machine). But Facebook's new model can take in the byte code as input and convert it to into a more efficient representation so that when this new code is converted into machine code, it will be faster. "In theory", it could mean that all existing code might be made faster without any change by developers. Our games can have higher frame rates on the same GPUs. Photoshop might run faster on the same laptop. Our software might complete tasks faster without any changes at all. But that will probably take quite a long time. To be honest I don't expect these models to actually be used in production since unless they were 100% as good as compilers they will probably break the code. And are they 100% good? No, not even close. But the speed at which improvements are happening, they will probably reach that level eventually. And that's when we will see models like these everywhere. 

11

There is a lot of interest in combining Graph Models and Language Models recently. In 2022, I led an initiative at Twitter (now X) in collaboration with Vivek Kulkarni, Ahmed El-Kishky, Sneha Mehta, and executed by our wonderful summer intern Jinning Li called "NTULM: Enriching Social Media Text Representations with Non-Textual Units" on combining large scale graph content at Twitter with Tweet's text content for building better quality Tweet Embeddings. The internal version of the project required addressing quite a few challenges: 1. Finding the node embeddings for NTUs using Twitter Scale Heterogeneous Graph 2. Training BERT with the above scale node embeddings This work was a follow up to my previous collaboration led by Vivek Kulkarni in 2021 on combining social context like time and geography in language models called "LMSOC: An Approach for Socially Sensitive Pretraining". There are a lot of interesting follow ups in this area and combining Graphs with generative LLMs is something which is very effective on real world datasets. NTULM Talk: https://lnkd.in/g2zvmnhh NTULM Paper: https://lnkd.in/gQq6VCQP LMSOC: https://lnkd.in/ghamVwVk

29

1 Comment

My teams' feedback on trainium chips from Amazon Web Services (AWS) - AWS's version of pytorch makes it such that we have to port model training done in the past for NVIDIA GPUs - that's a non-starter. If AWS wants more folks to use their chips for training/inference they should be first byte-for-byte compatible with pytorch used widely across the industry to train on NVIDIA. We would be okay, with some magic underneath the covers but the interface to pytorch can't change one bit.

19

5 Comments

📝 Exciting news! My latest blog post on Medium is now live! In this multi-series blog, I explore anomaly detection in any system. I share my experiences in designing and implementing an end-to-end solution for anomaly detection patterns in user access patterns. This series will cover a variety of topics, including understanding application events, optimizing data in Snowflake, developing machine learning models, and integrating with external applications. In the first blog post, I dive into everything about events and how they should be handled in a distributed system. Check out the full article here to learn more: https://lnkd.in/g8mjARdd I would love to hear your thoughts and insights in the comments section! Let's start a conversation. #events #eventing #outboxpattern #anomalydetection #machinelearning #dataoptimization #AWS #Snowflake

63

5 Comments

Last November the news was on fire, with speculation that an OpenAI model called Q* was getting close to AGI. Today Adam Selipsky announced their new product Amazon Q, coincidence? I think not. This used to be a product called CodeWhisperer. But it looks like it's merged into a product suite under the name Q. The website for CodeWhisperer had an announcement listed: "Amazon CodeWhisperer is now Amazon Q Developer, which is generally available. All of the functionality that CodeWhisperer provides is part of Amazon Q Developer – in-line code suggestions, security scanning, open-source license tracking, and more. This website will remain live until mid May. For the latest information and to learn more about Amazon Q Developer, visit the website." Any how, it looks like the play is to connect to internal data sources and employees to make generative AI apps based by their own apps. They note in their FAQs "When you sign up for Amazon Q Developer Pro or Amazon Q Business plans, we don't use content to improve underlying models for others. Your data is your differentiator." But it doesn't look like there are separate instances of the model but rather a super-model (yeah I made that pun). So if you trust that your data won't get integrated than you should be fine. Just like the trainwreck you couldn't look away from on the news. 🙄 P.S. 👍 If you found these insights useful, please like this post to support more content like this! 🔁 Feel free to share with your network. P.P.S. Subscribe to my newsletter, The Artificially Intelligent Enterprise, for weekly insights into how to succeed with AI in Business - https://lnkd.in/eqJNArYx

211

17 Comments

🚀 Excited to share my latest blog post on "Fine-tuning Meta's Llama3 with ORPO on Amazon SageMaker!"(https://lnkd.in/dP8VCbNR) 🦙 In this detailed guide, I explore the use of the new Odds Ratio Preference Optimization (ORPO) algorithm to streamline the Supervised Fine-Tuning (SFT) and preference alignment stages into a unified process. 👉 Dive into the blog for a step-by-step tutorial and access the full notebook (https://lnkd.in/dP_6GRjH) to fine-tune your own Llama 3 models. 🔍 To learn more about other fine-tuning and LLM alignment techniques, I'd highly recommend the online course Generative AI with LLM's by Shelbee Eigenbrode, Antje Barth, Chris Fregly & Mike Chambers (https://lnkd.in/drUPw2ud). In a future blog post I will be importing the model using Amazon Bedrock's newly announced custom model import feature for downstream empirical evaluation. P.S. For useful GenAI infographics and funny posts, check-out Eduardo Ordax's page. #LLMs #AI #Amazon #SageMaker #Meta #Llama3 #ORPO

56

3 Comments

Over last few months, I am increasingly leveraging LLMs to write code. From 10% to nearly 78% currently (based on lines of code) of my code is from LLMs. I have my own pipeline to fine tuned LLMs to augment my needs. Observations - 1. I am producing more code --> MORE IMPACT (approx 2.3X) 2. I am reading more code --> MORE MAINTAINABLE CODE 3. I am learning more as now I can start from concept and expand out ---> MORE LEARNING 4. I am incorporating best practices --> BETTER CODE 5. Lesser typing --> LOWER REPETITIVE STRAIN INJURY #AIBUDDY

76

8 Comments

The more research I read and the more I interact with LLMs, the more I wonder if some issues with combinatorial logic could be addressed by creating cyclic decoder-only models with a gating function for output. This approach might also enable a dramatic reduction in overall parameter count requirements for comparable performance. My understanding is that as model depth increases, higher-level abstractions form in the upper layers while some lower-level details also need to be propagated for use in the output token distribution. It seems larger models are effectively "unrolled" and need a fairly wide top to retain concept density as depth increases. By cycling a smaller graph, concepts and relations can intermingle more easily. If more layers of the smaller model are combined as input to generate the token distribution, they can specialize, reducing the need to propagate as much information up the graph for similar quality. At least, that's my hypothesis. The cycling could have either a learned gating function and/or a set number of cycles. I saw one research paper that seemed to touch on some of concepts, but I can't find it now. If anyone knows of other existing papers related to this idea, please let me know. If I've learned one thing, it's that there are almost no original ideas in model architecture. :)

9

Agentic AI is revolutionizing the industry of legacy code migration and modernization where we see clear positive business and societal impacts of this disruptive technology.

6

Microsoft just casually shared their new Phi-3 LLMs less than a week after the Llama 3 release. Based on the benchmarks in the technical report (https://lnkd.in/gtnfVEWr), even the smallest Phi-3 model beats Llama 3 8B despite being less than half the size. Phi-3 has "only" been trained on 5x fewer tokens than Llama 3 (3.3 trillion instead of 15 trillion) Phi-3-mini less has "only" 3.8 billion parameters, less than half the size of Llama 3 8B. Despite being small enough to be deployed on a phone (according to the technical report), it matches the performance of the much larger Mixtral 8x7B and GPT-3.5. (Phi-3 mini can be quantized to 4-bits, so it only requires ≈ 1.8GB of memory.) What is the secret sauce? According to the technical report, it's dataset quality over quantity: "heavily filtered web data and synthetic data". Next to the 4k context-window version, there's also a phi-3-mini-128K model that supports up to 128k tokens. Fun fact: Phi-3 uses the same tokenizer with a vocabulary size of 32,064 as Llama 2.

1,637

79 Comments

📣 Good folks at Apple have quietly dropped Core ML Gallery Models 🍏 Core ML is Apple's take on running on-device models using Apple Silicon, optimized for memory footprint and power consumption. 💾🔋 Models: https://lnkd.in/gFCSDMVn These models are ports/quantized versions of PyTorch models 🔄🧠 HF Core ML examples for a quick start: https://lnkd.in/g7PXSh_n 🚀

16

1 Comment