Swapnil Ghike

Test time training of LLM. For easier problems providing context along with the prompt is enough. For harder problems like math, coding, law etc. taking a sample of similar problems and training few batches on it improves accuracy many X times (3-14 times or so). https://lnkd.in/gwp97RFr

1

I believe a solution to this is currently being developed.

5

𝐎𝐧 𝐭𝐡𝐞 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐭 𝐡𝐮𝐞𝐬 𝐨𝐟 "𝐈𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐓𝐢𝐦𝐞 𝐒𝐜𝐚𝐥𝐢𝐧𝐠" (𝐈𝐓𝐒) 𝐢𝐧 𝐋𝐋𝐌𝐬. Since the release of 🍓 o1, Inference Time Scaling (ITS) has become a new buzzword, with some previous techniques being rebranded as instances of the current ITS buzz.. The most basic thing common about all the uses of ITS seems to be that at inference time, LLMs are made to do more than just output the completion of the prompt (thus taking more compute at the inference time than standard autoregressive LLMs). The specific way it is done varies quite a bit and impacts the accuracy. Here is a small taxonomy of the various brands of ITS--with some comments about their relative power. 1. 𝗖𝗵𝗮𝗻𝗴𝗶𝗻𝗴 𝗼𝗻𝗹𝘆 𝘁𝗵𝗲 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 (𝗸𝗲𝗲𝗽𝗶𝗻𝗴 𝘁𝗵𝗲 𝗽𝗿𝗲-𝘁𝗿𝗮𝗶𝗻𝗲𝗱 𝗟𝗟𝗠 𝘁𝗵𝗲 𝘀𝗮𝗺𝗲) The general idea here is to sample multiple candidates and select the response from them. There are two issues: How to generate a large sample of candidates and, how the promising candidate is selected [Candidate Generation:] The generation of multiple candidates can be done either by sampling directly from the LLM (with appropriate temperature settings), or augmenting the original prompt with additional augmenting strings to generate additional candidates. These augmenting strings have originally become popular with the "Chain of Thought" papers--but by now it is clear that augmenting strings don't have to be anthropomorphized as "chains of thought"! (c.f. https://lnkd.in/gCCPGz-e). If we are considering augmenting strings (that we will call CoT strings--without any anthropomorphization), it is also possible to search among multiple CoT strings. (the recent Marco-o1 seems to do this explicitly). [Selection:] Once the candidates are generated, the promising one(s) need to be selected. The approaches here include 1.1 Simple majority voting (and other versions such as self-consistency) 1.2 Verifier based selection from the generated candidates. [1.2.1 LLMs as verifiers] Some early methods used LLMs themselves as verifiers--but there is work showing that LLMs are no better at self-verification than generation in the first place. They can have both false positives and false negatives. A crucial corollary is that the candidate eventually selected could still be incorrect--which would be problematic especially in cases where deployment failures are costly (c.f. https://lnkd.in/grv6DgYu) [1.2.2 Learned Verifiers:] It is possible to improve the verification accuracy by learning task-specific verifiers; this has received some traction under the name "generative verifiers"--but it is a rather old idea that aims to leverage the fact that learning to predict correctness of the candidate is a "discriminative task" and thus can be of lower sample complexity (c.f. https://lnkd.in/gbMhcVtP). The learned verifiers can improve accuracy but provide no guarantees. (Continued in the comments below)

97

8 Comments

Here's the technical breakdown of the AI Penalty Challenge authored by Alok Pattani (check out this video if you haven't seen the demo yet: https://lnkd.in/gsD3xjJ2) Key Takeaways: ☁️ Firestore for triggering slow-mo video capture on Android phones & uploading to Google Cloud via Firebase Storage. 🧠 Cloud AutoML for training custom object detectors (ball & goal tracking) to calculate speed & accuracy. 🗣️ Gemini Pro 1.5 on Vertex AI for personalized coaching feedback based on multiple data points. 🎨 Imagen on Vertex AI for instant, prompt-driven image editing for cool player pics. More technical insights from the project to come... #AIChallenge #MachineLearning #SportsTech #GoogleCloud #footballNOTsoccer

89

5 Comments

Purely for LLMs to do maths and logic will always be in a realm of non determinism. If we give it a problem which needs say solving constraints (as an example a Sudoku), there is always a chance of model not giving accurate results going by the auto regressive nature of it. For such problems We can still couple a symbolic verifier like a Z3 system to LLM to get best of both worlds. I created a Claude MCP server backed by a Z3 solver and hooked it to Claude. Given a problem which reauires constraints solving (like scheduling, Sudoku), Claude generates Z3 specific code and invokes the Z3 MCP Server to validate the results.. This looks to be a good direction towards building Neuro Symbolic Systems, by coupling task specific MCP servers (like github,filesystem, python interpreters etc)

2

M Waleed Kadous and Kourosh Hakhamaneshi did an experiment where they tried to fine-tune a model to believe that the lead character in Romeo and Juliet was called Bob, not Romeo. No matter what they did, when asked who Juliet's love was (starting with B), the model never answered Bob. Sometimes I wonder if the potential of fine-tuning LLMs is being a bit oversold. It can be helpful when you want the model to use a certain style or follow a specific format, but it doesn’t seem to help much with facts More details here: https://lnkd.in/g6p2eU-n

8

🔍 Why Chunking Strategy is Key in Building a RAG System with LLMs 🔍 Recently, a student of mine while attending the interview was asked to explain how he would chose chunking strategy to build a better RAG system. When building a Retrieval-Augmented Generation (RAG) system using LLMs, chunking is a critical step. The process of splitting data into manageable pieces not only affects how well relevant information is retrieved but also impacts the LLM’s ability to generate coherent and accurate responses. • Contextual Understanding: LLMs have limitations in processing extremely long texts. Chunking helps maintain context and ensures the model can focus on relevant information. If you are using Opensource LLM with limitation on context length, this becomes important.  • Efficiency: Smaller chunks reduce the computational burden on the LLM, leading to faster responses. • Retrieval Accuracy: Chunking can improve the accuracy of information retrieval by making it easier to find relevant segments. Essentially, Chunks should not be too large or too small. So, what’s the right answer ? Well, it depends on the problem you are solving. First you should ask below questions to yourselves. • What is the nature of the content being indexed? Long documents, Short sentences, Paragraphs ? • Which embedding model are you using, and what chunk sizes does it perform optimally on? • What are your expectations for the length and complexity of user queries? Will they be short and specific or long and complex? This may inform the way you choose to chunk • How will the retrieved results be utilized within your specific application? For example, will they be used for semantic search, question answering, summarization, or other purposes? Then chose one of the below strategies: 𝗖𝗼𝗺𝗺𝗼𝗻 𝗖𝗵𝘂𝗻𝗸𝗶𝗻𝗴 𝗦𝘁𝗿𝗮𝘁𝗲𝗴𝗶𝗲𝘀: Sentence-Based Chunking Paragraph-Based Chunking Token-Based Chunking Overlapping Chunking (Sliding Window) Topic-Based Chunking: Identify topic boundaries using techniques like topic modeling or keyword extraction. Specialized chunking : You would need specialised packages for chunking if you are using LaTex or Markdown etc. Take that into consideration. Semantic Chunking : Variable length chunking retaining the semantic coherence. Creating Sentence groups etc. Langchain has a Semantic Chunking splitter. Hybrid Chunking: Combine multiple strategies to optimize for your specific use case. Choosing the Right Strategy: The best chunking strategy depends on factors like the structure of your documents, the complexity of the information, and the desired level of granularity. Experimentation is often key to finding the optimal approach. By carefully considering chunking strategies, you can significantly enhance the performance and effectiveness of your RAG system.

7

1 Comment

🚀 Day 18 of My 25-Day Generative AI and LLM Challenge 🤖🚀 🌟 How Variational Autoencoders (VAEs) Work: Encoding & Decoding 🌟with my friends HARSH JAVIA and Maanan Jagani! Variational Autoencoders (VAEs) are a type of generative model that aim to learn the underlying structure of data. Here's a simplified breakdown of how they work, using a bit of professional flair and emojis to keep it engaging! 🔍 1. Understanding the Basics🔍 At a high level, VAEs consist of two main components: 1. Encoder 🧠: - Takes input data (e.g., images, text) and compresses it into a lower-dimensional space known as the **latent space**. - Think of this as summarizing the essence of the input data into a compact representation. 2. Decoder 💡: - Takes the encoded representation from the latent space and tries to reconstruct the original input data. - This process helps the model learn to generate new data that is similar to the input data. 🚀 2. Step-by-Step Process 🚀 Step 1: Encoding 📥 - The encoder network receives the input data and transforms it into a mean (μ) and a standard deviation (σ). - From these, a latent vector \( z \) is sampled using the reparameterization trick: \( z = \mu + \sigma \cdot \epsilon \), where \( \epsilon \) is a random variable from a standard normal distribution. Step 2: Latent Space 🌌 - The latent space is a lower-dimensional representation that captures the essential features of the input data. - This space allows the model to explore various possible data representations. Step 3: Decoding 📤 - The decoder network takes the latent vector \( z \) and attempts to reconstruct the original input data. - The quality of the reconstruction indicates how well the model has learned the data's underlying structure. 🌟 3. Why VAEs are Powerful🌟 - Data Generation 🎨: VAEs can generate new, unseen data that is similar to the training data, useful in creative applications like art and music. - Dimensionality Reduction 📉: They reduce the complexity of data, making it easier to visualize and analyze. - Anomaly Detection 🔍: VAEs can identify anomalies by recognizing data that doesn’t fit the learned patterns. 🏆 4. Professional Takeaway 🏆 Variational Autoencoders are a powerful tool in the machine learning toolbox, enabling efficient data compression, reconstruction, and generation. Their ability to model complex data distributions makes them invaluable for various applications, from image generation to anomaly detection. #MachineLearning #AI #DataScience #DeepLearning #GenerativeModels #VariationalAutoencoders #VAEs #NeuralNetworks #DataInnovation #TechTrends #ArtificialIntelligence #MLApplications #DataGeneration #TechInspiration #Innovation #AIResearch #DataAnalytics #DeepLearningModels #TechCommunity

6

This benchmark data is being used in various LinkedIn posts selectively to make a specific argument about language choice while ignoring several critical factors: 1. Single-metric fallacy: Using just one artificial benchmark (nested loops) to argue for wholesale language replacement ignores the many other factors that influence language choice in real projects, such as: - Ecosystem and library availability - Team expertise and hiring pool - Development speed and maintenance costs - Existing codebase compatibility - Specific project requirements 2. False equivalence: The post suggests that Python's type annotations make it "weird" and therefore one should switch to Rust. This conflates two separate concepts: - Python's optional type hints are development tools for code clarity and IDE support - Rust's compile-time type checking is part of its memory safety system These serve different purposes and aren't directly comparable. 3. Context omission: The benchmark fails to mention that: - Most real Python code uses vectorized operations through NumPy/Pandas - Production Python code rarely relies on raw nested loops - Python's strength lies in its extensive data science/ML ecosystem, not raw computation - Many Python applications use C extensions for performance-critical parts 4. Premature optimization: For many Python applications, the execution speed of nested loops is not the bottleneck - factors like I/O, database access, or network latency are often more significant. Choosing a language solely based on loop performance could be a case of premature optimization. A more balanced approach would be to evaluate languages based on their fitness for specific use cases, rather than using a narrow benchmark to make broad claims about language superiority.

2

1 Comment

https://lnkd.in/gqUkq3BM is an excellent overview of major topics in LLMs. If you are getting started in this space, this video which is also generated by another LLM 😂 , is a great place to begin your journey in this exciting area of AI. It covers topics like pre-training, evolution of GPTs and other popular LLMs, fine tuning (SFT, PEFT like LORA, RLHF), inference optimizations (quantization, speculative decoding, distillation, flash attention), output sampling techniques, multimodal capabilities and finally some real world applications like math, coding, medicine etc. Loved their analogies even for core complex concepts like flash attention - "it's like finetuning the engine of a car, to make it more fuel efficient, without changing how the car drives"

24

User experience( New User Interactions), Accuracy and Interpretability - Technology Handicap that makes LLM yet another upgrade of Search Engine. LLMs are hugely successful in terms of User engagement and exponentially increasing User expectations in terms of accuracy on first interaction. But for OpenAI trying to be search Engine and again giving options for User to filter out from set of results. The idea should have been to Just give User the information without Users having to visit any links. Even Meta launched AI search Engine. The problem really is, are LLM able to replace the trust of the brands that have created those content ? what are the ways in which generated text can be annotated by brand tags. While Training LLM has really lost the information of source from which the info was Learnt. Thus the only option is to again query the output vectors(generated text)to search for nearest links. Model Interpretability can solve for text annotation and build trust. Next interactions of User with long and complex query cannot be yet another link that may take away all the comprehension and summarization, next interaction needs to be an action (Booking, Purchase, Order, create PPT, Send email, Generate report) Giving links is a lost cause....

1

1 Comment

I remember when this all started. Faris - invited me to catch up at our favorite spot around Blok M. I had no idea he’d just wrapped up a procedure at Pondok Indah Hospital! (He really knows how to add drama to our catch-ups.) We’d been on separate paths and projects since "the winding down project", but there he was, reminding me of our dream to ride this AI wave together again. It was a plan we’d hatched with Sahal Zain & Gurnoor Dhillon when the world of AI was still mostly theoretical and jargon-heavy. Somehow, the conversation continued with Gurnoor Dhillon while he was waiting for his meeting with someone who would eventually become one of CalvinBall’s key early adopters. It felt like everything was coming together, like a series of seemingly random dots connecting to form a bigger picture. Seeing how far we’ve come as a team at CalvinBall Technologies is truly exciting. What started as theoretical discussions has now evolved into practical Gen AI solutions that are solving everyday problems and hold endless possibilities to shape the industry. 👨‍🚀 👩‍🚀 Here’s to pushing boundaries and achieving new milestones together! Gurnoor Dhillon, Sandeep Ramesh, Faris -, Sahal Zain, Eva Zuliya, Ratu Nadiah Khairunnisa 🚀 #GenAI #AIInnovation #TechJourney #ArtificialIntelligence #TechTransformation #CalvinballTech #Calvinballers

55

2 Comments

No amount of scaling (throwing billion$ for computational resources & trillions of words) would help LLMs achieve the two key cognitive capabilities that make human intelligence so special: 1. Learn - Autonomously, Adaptively, Conceptually, Contextually & Incrementally. 2. Reason - Conceptually, Contextually & Metacognitively We discussed the criticality of this in greater detail here: https://lnkd.in/ggZm_W-w It is for this reason, LLMs can never get us to AGI: https://lnkd.in/d3vEPJ2P So, what will get us to AGI: https://lnkd.in/ggFTxuUz It's no wonder Demis Hassabis, CEO of Google DeepMind said this about LLMs Intelligence on July 9, 2024 at the Future of Britain Conference hosted by Tony Blair: “We’re still not even at cat intelligence yet, as a general system,” https://lnkd.in/gtyQNZyc It is great to see Yann LeCun arriving at the same conclusions about LLMs. https://lnkd.in/gm9Nfe4P

342

68 Comments

Looking forward to this panel where Divakar Tantravahi, Meduri Ravi Kumar, and Cosimo Spera will talk about the latest AI technologies that are changing product and customer support, along with their real world examples and practical strategies. If you have not registered, use this link - https://lnkd.in/gdTMgryA #ai #businessoutcomes #product #customer

22

Honored to be featured in a thought-provoking article by Jayesh Shinde for Digit.in, one of India's leading tech magazines, on the rise of Small Language Models (SLMs) in AI! 🇮🇳🚀 A huge thank you to Jayesh and the Digit team for highlighting our work at Katonic AI in this crucial conversation. As I shared in the article, SLMs are truly revolutionizing the AI landscape: • 60-70% faster development cycles • Comparable performance to larger models in specific domains • Cost-effective AI solutions for a broader range of businesses At Katonic, we're not just talking about the potential of SLMs - we're making it a reality with our Katonic Adaptive Studio. Our platform enables: 🔒 Complete data sovereignty and control 🎯 Cutting-edge model tuning capabilities ⚡ Lightning-fast inference and optimized resource utilization We're seeing firsthand how this approach is transforming businesses, with up to 50% reduction in operating costs compared to building AI clusters from scratch. The future of AI isn't just about bigger models - it's about smarter, more efficient ones that can be deployed securely within your own environment. Curious about how SLMs and adaptive AI can benefit your business? Let's connect and explore the possibilities! #ArtificialIntelligence #SLM #AIInnovation #EnterpriseAI #KatonicAI #MakeInIndia Digit(AI)zed

44

4 Comments

Lot of folks are confusing "bots" with "agents". Agents without "stop and think" capability are not really agents. This means agents are not simple sequential step-wise operator in a defined flow. They may be classified more as bots. Agents take goals, instructions in unstructured format like conversational inputs, plan the tasks around goals (stop and think) and then act on those tasks. Bot example (can be programmed as "if else" statement using coding): > Label my email as "report" if it comes from [email protected] Agent example (needs thinking, planning and action): > Send me Whatsapp if I receive any email which is important but avoid my meeting hours unless it is from my boss Always look at language, planning and action planes in Agent wireframe! Always look at instruction layer, orchestration layer and tooling layer in the agent stack. Agent would need a conversational interface (eg. chatbot) and enable automation of a process for which tool integration will be key. Otherwise it may just be a "if-else bot" disguising as agent! They just have action planes.

35

2 Comments

I am excited to share the latest blog, co-authored by myself, Junhwi Kim and Vyas Sekar, detailing how we managed to successfully localized and improved LLM by combining Fine-Tuning and RAG. https://lnkd.in/gKDkgGPq

70

3 Comments

Take a look at a Flight Discovery Chatbot that I built few days back to help fellow travelers plan their trip with the power of LLM and Multi-Agentic RAG Autogen Workflow in Python leveraging Natural Language processing. What-if a chatbot can answer your queries and help you in finding cheapest and fastest flights and aid you in planning an efficient trip?That’s what I aimed to do with my this LLM project which leverages Multi-agentic workflow to aid flight discovery. This side weekend project digs into Natural Language Processing (NLP) which harnesses the power of AI to understand, interpret, and generate human language, transforming how we interact with technology. Autonomous agents, equipped with AI capabilities, independently perform tasks and make decisions, enhancing efficiency and scalability across various applications. Last year, Large Language Models (LLMs) were the buzzword, capturing widespread attention for their revolutionary advancements in natural language processing. Today, however, the focus has shifted to multi-agent workflows, which are now at the forefront of innovation, promising enhanced collaboration, efficiency, and scalability in AI-driven solutions. In this blog I share the details of the workflow that I built with components like Autogen, Amadeus Flight API's Developer Free Tier for Developers and Neon Serverless Postgres Database. #AI #LLM #Autogen #Chatbot #Agentic #RAG #technology #AIML #NLP #MachineLearning

33

3 Comments