Saugata Paul

Training LLMs using reinforcement learning using human feedback. I was slacking on this topic for 6 months and finally was able to go through it, and wow I am amazed at what technologies are being worked on in production-ready systems these days. Generally in RL of LLMs, we have a reward model (let's say a good/bad text classifier) that gives a reward (prediction 1 for good, -1 for bad) and an RL algorithm (based on multiple loss functions) updates the weights of the LLM model to achieve better results. This is an iterative process between the reward model, the LLM, and the RL algorithm. Overall, my take is the most important aspect of training LLM using RLHM is obtaining that human feedback and training a reward model.

28

🚨Paper Alert 🚨 ➡️Paper Title: Loong: Generating Minute-level Long Videos with Autoregressive Language Models 🌟Few pointers from the paper 🎯It is desirable but challenging to generate content-rich long videos in the scale of minutes. Autoregressive large language models (LLMs) have achieved great success in generating coherent and long sequences of tokens in the domain of natural language processing, while the exploration of autoregressive LLMs for video generation is limited to generating short videos of several seconds. 🎯In this work, authors conducted a deep analysis of the challenges that prevent autoregressive LLM-based video generators from generating long videos. Based on the observations and analysis, authors have proposed “Loong”, a new autoregressive LLM-based video generator that can generate minute-long videos. 🎯Specifically, they modeled the text tokens and video tokens as a unified sequence for autoregressive LLMs and trained the model from scratch. They proposed progressive short-to-long training with a loss re-weighting scheme to mitigate the loss imbalance problem for long video training. 🎯They further investigated inference strategies, including video token re-encoding and sampling strategies, to diminish error accumulation during inference. 🎯Their proposed Loong can be trained on 10-second videos and be extended to generate minute-level long videos conditioned on text prompts, as demonstrated by the results. 🏢Organization: The University of Hong Kong , ByteDance 🧙Paper Authors: Yuqing Wang, Tianwei Xiong, Zhou D., Zhijie Lin, Yang Zhao, Bingyi Kang, Jiashi Feng , Xihui Liu 1️⃣Read the Full Paper here: https://lnkd.in/gcE_-5wF 2️⃣Project Page: https://lnkd.in/gU3rU6vU 🎥 Be sure to watch the attached Demo Video - Sound on 🔊🔊 Find this Valuable 💎 ? ♻️REPOST and teach your network something new Follow me 👣, Naveen Manwani, for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements.

9

How are Graph Convolutional Neural Networks (GCNNs) a generalisation of Convolutional Neural Networks (CNNs)? My next blog in the Graph series on "Deep Learning with Graphs" is on the way (well, it's already written on paper) and it's going to be a blockbuster. The beauty of Graph Machine Learning is that the more you read about it, the more you feel that your neurons (real ones!) are learning to think better! And you're already in love with it! Coming to the point: One might recall the convolution operations in a CNN. You have a grid of pixels representing an image. You have a 3 x 3 filter. You slide the filter over the image and carry out aggregation to obtain a new pixel value. You proceed in the same fashion over the whole grid, sliding towards the right , bottom, carrying out the convolution operation and finally getting a new transformed, reduced image through the series of aggregations. You repeat this through the different convolutional layers which together form a Convolutional Neural Network Now think of a Graph. It is not simple grid like an image. The topology of a Graph is arbitrary. There is no notion of left, bottom, right in a Graph like in an image which is represented through a grid of pixels. You cannot slide over a Graph as simplistically you did and performed the convolutional operations in an image. How do you approach the problem? How do you carry out convolutional operations in a Graph? Now take a node in the Graph. Take all the nodes which are the immediate neighbours of this node. The convolutional operation here is: You aggregate the information from all the neighbours and pass it to the node. The information is nothing but the emedfings of the neighbour nodes. You add together the emedding of the node under consideration. You transform the aggregated embedding by multiplying with a weight Matrix and transform the node's own embedding by multiplying with a different weight matrix. This is the typical linear transformation operation in a NN. You apply non linearity - the activation function like ReLu. Now think about a CNN taking the same grid and considering a central pixel. Consider the rest of the pixels around the central pixel. You are essentially doing the same thing: taking information from the neighbour pixels along with the central pixel, aggregating, as I described above for the node in the Graph. Hence a Graph Convolutional Network is a generalisation of CNN The story goes, you do the same for every node on the Graph. Each node in the Graph represented through a different neural network architecture! This is amazing 🔥🔥🔥💓💓💓 #graphs #machinelearning #deeplearning My Technical Blogs Library: 1) https://lnkd.in/ejyWacXn 2) https://lnkd.in/bfrkRjq Follow Ajay Taneja for insights on: Data Science, Data Engineering, Finite Element Analysis, Non Linear Finite Element Analysis, Fatigue, Fracture Mechanics

22

Here's my blog on the evolution of attention mechanisms. From Bahdanau's additive attention (2014) to Luong's multiplicative attention (2015) and Vaswani's self-attention in Transformers (2017). Blog:- https://lnkd.in/guWW5Eyb

31

2 Comments

Many folks have asked me what are the prerequisites to starting ML. There are some must-have and good to have pre-requisites that should/can be done before starting ML Here's my list Must have Linear algebra: vector spaces, sup-spaces, linear transformation, spectral decomposition, SVD, eigenvalues and eigenvectors Probability: fundamentals of distribution, PMF, PDF, some continuous and discrete distributions like Gaussian distribution Calculus: high school calculus. Information theory: the idea of entropy and mutual information Good to have Linear Algebra: operator and linear products, theory of solutions Probability: stochastic processes, moments and generating functions, Markov process Calculus: optimization under constraint, jacobian and Hessian, lagrangians Information theory: channel theory, information bottleneck

29

1 Comment

Post 1/2: General guidelines for your ML Technical interviews: Before Applying: Research the company and team in advance. Check out their blogs and the prospective team members. Try to align and highlight your skills and key projects that are aligned to the company or the team or the role you are applying to. For example: Let's say you have have completed a variety of ML projects in both computer vision and NLP. If you are applying for an LLM engineer role, you might want to highlight your courseworks, projects, etc. related to NLP. Try to highlight those libraries that you have used for such projects. While writing CVs, try to write concisely in 1-2 lines and use bullet points if possible, instead of writing long sentences and paragraphs. Don’t Inflate your skills and contributions: I have noticed that a few candidates mention their group projects, without having core contributions and passing it as their own, while their contributions are almost marginal. During interviews, when asked about such group projects and their details, the candidates bail out saying they don’t know because they were not working on that particular aspect and they had worked on other aspects. This may bring out a bad impression to your interviewer. Now, the question is should you include or not such projects in your CV? Personally, I think it is fine to add such projects, as long as you have good understanding and you are able to explain the underlying concepts and all the aspects of the projects, while clearly mentioning your contributions. Putting lots of jargon and keywords: Your resume might get noticed in the initial stages, when someone might be searching for potential candidates and the keywords in your resume might get you lucky. However, on the long run they might hurt The more keywords you add and highlight, there is a high chance the interviewer may assume that you have thoroughly understood or applied those concepts. Unless you are prepared to explain those concepts in detail along with their intuition and understanding, I would suggest not using unnecessary jargon. Continue reading in Post 2/2... https://lnkd.in/gQGjMGJ9 P.S. This post does not generalize the interview process and varies from interviewer to interviewer. Each organization may have their own interview policies and may have their own style of interview process. #machinelearning #interviews #interviewpreparation #interviewtips

60

2 Comments

Neural networks are frequently employed to solve Partial Differential Equations (PDEs) across various domains. However, nonlinear PDEs with multiple solutions present a significant challenge for existing neural network techniques. Function learning strategies attempt to learn the solution function but often falter due to an ill-posed problem. Conversely, operator learning methods, such as Physics-Informed Neural Networks (PINN) and DeepONet, aim to approximate the mapping between parameters and solutions. Introducing NINO: a novel method designed to tackle nonlinear PDEs with multiple solutions through operator learning techniques. NINO enhances traditional Newton methods by integrating them into a superior network architecture and formulating problems more effectively. NINO surpasses conventional Newton methods and neural operator techniques in terms of execution speed. It employs two distinct training methodologies: one using Mean Squared Error Loss and another that combines supervised and unsupervised learning. In experiments, the neural operator method efficiently learned the Newton operator, requiring minimal supervised data. https://lnkd.in/edM5VRWh

1

Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters Exploiting activation sparsity helps accelerate inference process of large language models (LLMs) without compromising performance. However, activation sparsity is determined by activation functions, and commonly used ones like SwiGLU and GeGLU exhibit limited sparsity. To address these challenges, this paper proposed a novel dReLU function, which is designed to improve LLM activation sparsity, along with a high-quality training data mixture ratio to facilitate effective sparsification. 𝗞𝗲𝘆 𝗰𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻𝘀: - Efficient dReLU activation function: proposed method utilizes fewer than 150B tokens, representing less than 1% of typical pretraining tokens (commonly 15T tokens). - Sparse activated models: released sparsely-activated TurboSparse-Mistral7B and TurboSparse-Mixtral-47B models, which shows better performance compared to original versions - Practical inference speedup: Evaluation shows that with released models, 2-5× speedup can be achieved, with up to 10 tokens/s even without a GPU on TurboSparse-Mixtral-47B. 𝗔𝗻𝗮𝗹𝘆𝘀𝗶𝘀 i) Limitations of existing ReLUfication - sparsity evaluation of ReLULlama-7B and original Llama-2-7B, shows that existing ReLUfication methods can only improve the sparsity from 40% to 67% - This can be attributed to the ReLU activation function applied after gate weight, which masks all negative activations to zero ii) dReLU - newly proposed activation function, where ReLU is applied after both the up- and gate-projection - for an experiment comparing models using dReLU and SwiGLU, results shows that dReLU structure exhibit similar convergence compared to those using SwiGLU -further evaluation results for various sparsity levels demonstrate dReLU activation function enables high sparsity without significant degradation in performance, maintaining competitive performance even at 90% sparsity - In contrast, the SwiGLU-based model experiences a severe increase in perplexity as sparsity reaches 80%. 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝗮𝗹 𝗥𝗲𝘀𝘂𝗹𝘁𝘀 - TurboSparse-Mistral-7B outperforms Gemma-2B by far, while only activating 3B parameters. - TurboSparse-Mixtral-47B outperforms the original Mixtral-47B with only 4.5B parameters activated - for activations with a value of zero, TurboSparse-Mistral-7B, on average, has 90% of the neurons inactive in each layer, and TurboSparse-Mixtral-47B with 85% on average for each expert FFN 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗦𝗽𝗲𝗲𝗱𝘂𝗽 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 - On Pure CPU Inference, ReLUfied models achieve 2.08-2.28× speedup over original model. - On Hybrid GPU-CPU Inference, demonstrate significant speedups ranging from 2.52 to 4.64× compared to the baseline llama.cpp - For Deploying LLMs on mobile phones, achieves a 22.2× speedup using TurboSparse-Mixtral-47B inference compared to llama.cpp with original Mixtral-47B 𝗣𝗮𝗽𝗲𝗿: https://lnkd.in/eeHCWhUt 𝗠𝗼𝗱𝗲𝗹𝘀: https://lnkd.in/etHpZUju

24

1 Comment

🚨ECCV 2024 Paper Alert 🚨 ➡️Paper Title: AMEGO: Active Memory from long EGOcentric videos 🌟Few pointers from the paper 🎯Egocentric videos provide a unique perspective into individuals' daily experiences, yet their unstructured nature presents challenges for perception. In this paper, authors have introduced “AMEGO”, a novel approach aimed at enhancing the comprehension of very-long egocentric videos. 🎯Inspired by the human's ability to maintain information from a single watching, AMEGO focuses on constructing a self-contained representations from one egocentric video, capturing key locations and object interactions. 🎯This representation is semantic-free and facilitates multiple queries without the need to reprocess the entire visual content. Additionally, to evaluate their understanding of very-long egocentric videos, they introduced the new Active Memories Benchmark (AMB), composed of more than 20K of highly challenging visual queries from EPIC-KITCHENS. 🎯 These queries cover different levels of video reasoning (sequencing, concurrency and temporal grounding) to assess detailed video understanding capabilities. They showcased improved performance of AMEGO on AMB, surpassing other video QA baselines by a substantial margin. 🏢Organization: Politecnico di Torino, FAIR, Meta, University of Bristol 🧙Paper Authors: Gabriele Goletto, Tushar Nagarajan, Giuseppe Averta, Dima Damen 1️⃣Read the Full Paper here: https://lnkd.in/gz_qj8Bz 2️⃣Project Page: https://lnkd.in/gEii6UtN 3️⃣Code: https://lnkd.in/gfBH-Eix 🎥 Be sure to watch the attached Demo Video - Sound on 🔊🔊 Find this Valuable 💎 ? ♻️REPOST and teach your network something new Follow me 👣, Naveen Manwani, for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements. #ECCV2024 #egocentricvision

60

1 Comment

🚀 What Happens to a Contextual Embeddings Model if a Word is Not Present in the Vocabulary? 🌟 In contextual embeddings models like BERT, GPT-3, etc., handling out-of-vocabulary (OOV) words is crucial for maintaining performance and accuracy. These models typically address OOV words by breaking them down into subword units using techniques like WordPiece or Byte-Pair Encoding (BPE). This means that even if the entire word isn't present in the vocabulary, its parts likely are. Here's how it works: 1. Subword Tokenization: When a word is not in the vocabulary, it is split into smaller subword units. For example, the word "newword" might be broken down into "new" and "word", or even into smaller units like "ne", "##w", "##word" in the case of BERT. This approach ensures that the model can still generate embeddings for parts of the word, capturing some of its meaning and allowing it to be processed effectively. 2. Retaining Semantic Information: By breaking down OOV words into known subwords, the model can retain and leverage semantic information from the subwords. This allows the model to handle a wide range of words, even those it hasn't seen during training. For instance, the subwords "new" and "word" in "newword" carry meaningful information that the model can use to infer the context and meaning of the OOV word. 3. Handling Complex Words and Morphology: Subword tokenization is particularly useful for handling complex words and morphological variations. It reduces the number of OOV words and improves the model's ability to generalize across different forms of words. For example, the word "unhappiness" might be broken down into "un", "##happi", and "##ness", allowing the model to understand its structure and meaning more effectively. 4. Contextual Understanding: Contextual embedding models like BERT and GPT-3 are pre-trained on large corpora, which helps them understand and generate embeddings for subwords based on their context. This allows them to handle OOV words dynamically and accurately. Even if a word is rare or newly coined, the model can use its context to generate meaningful embeddings and produce relevant responses. By leveraging subword tokenization techniques, contextual embeddings models can effectively manage OOV words, ensuring robust performance and accurate language understanding. Stay tuned for more insights on NLP and AI advancements! 💡 #NLP #AI #MachineLearning #DataScience #LanguageProcessing #TechInnovation #BERT #GPT3 Please follow my posts for more updates: Tejas Nanaware on LinkedIn #machinelearning #deeplearning #artificialintelligence #datascience #statistics Feel free to connect and share your experiences or ask any questions!

5

🚨ACL 2024 Outstanding Paper Award Alert 🚨 ➡️Paper Title: INDICLLMSUITE: A BLUEPRINT FOR CREATING PRE-TRAINING AND FINE-TUNING DATASETS FOR INDIAN LANGUAGES 🌟Few pointers from the paper 🎯Despite the considerable advancements in English LLMs, the progress in building comparable models for other languages has been hindered due to the scarcity of tailored resources. 👁In this paper Authors work aims to bridge this divide by introducing an expansive suite of resources specifically designed for the development of Indic LLMs, covering 22 languages, containing a total of 251B tokens and 74.8M instruction-response pairs. 🎯Recognizing the importance of both data quality and quantity, their approach combines highly curated manually verified data, unverified yet valuable data, and synthetic data. 🎯They built a clean, open-source pipeline for curating pre-training data from diverse sources, including websites, PDFs, and videos, incorporating best practices for crawling, cleaning, flagging, and deduplication. 🎯For instruction-fine tuning, they amalgamate existing Indic datasets, translate/transliterate English datasets into Indian languages, and utilize LLaMa2 and Mixtral models to create conversations grounded in articles from Indian Wikipedia and Wikihow. 🎯Additionally, they addressed toxicity alignment by generating toxic prompts for multiple scenarios and then generated non-toxic responses by feeding these toxic prompts to an aligned LLaMa2 model. 🎯 They hope that the datasets, tools, and resources released as a part of this work will not only propel the research and development of Indic LLMs but also establish an open-source blueprint for extending such efforts to other languages. 🏢Organization: 🛎️Nilekani Centre at AI4Bhārat🛎️ Indian Institute of Technology, Madras 🛎️Microsoft🛎️Sant Longowal Institute of Engineering and Technology 🛎️Indian Institute of Information Technology Design & Manufacturing Kancheepuram, 🛎️Maharaja Agrasen Institute Of Technology, Delhi 🛎️Sarvam ,🛎️ NICT 🧙Paper Authors: Mohammed Safi Ur Rahman Khan, Priyam Mehta , Ananth Sankar , Umashankar Kumaravelan, Sumanth Doddapaneni, Suriyaprasaad G, Varun Balan, Sparsh Jain, Anoop Kunchukuttan, Pratyush Kumar, Raj Dabre, Mitesh Khapra 1️⃣Read the Full Paper here: https://lnkd.in/gqwQST-r 2️⃣Code: https://lnkd.in/giWPVntx 🎉 🎉  Congratulations to all the authors 🎉 🎉 Find this Valuable 💎 ? ♻️REPOST and teach your network something new Follow me 👣, Naveen Manwani, for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements. #ACL2024 #Indicllms #llms #llama2 #mixtral

16

Unlike software engineering or software development the entry barrier for a machine learning engineer is still very high and it hasn’t changed much over the last 5 years Being an ML engineer at FAANG still requires one to have a masters or PhD. So how can you enter this field? I interview for Amazon’s applied scientist post and cleared 3 rounds. Even though I was just a bachelors degree holder I got shortlisted and performed well for the 3 rounds. So how did I do it? Why was I shortlisted? Is it because I’m from tier 1? Not necessarily, cause I did have a talk with Amazon employees as to what it takes to be an applied ML scientist at Amazon. Having a masters or phd surely increases your chances but here’s how you can get shortlisted and probably selected as well for an applied ML scientist position at Amazon or other FAANG or bit MNCs. The first important thing to have is, prior work experience to prove that you’re proficient with ML. It can either come through internships, projects or publications. Fortunately I had them all from my undergraduate studies, so it increased my chances. Having a good kaggle rating is really really important if you don’t have internships or projects to write. Very few people do kaggle regularly and it’s really a game changer. Even I didn’t do kaggle much in college. Practice maths behind ML is I cannot emphasize how important maths is. Most of the students I talked to, when I ask them why they used a certain model or certain type of architecture or regularizer for this problem stumble to give me any strong answer cause their solution was not mathematically motivated but rather something they found on tutorials or internet. It’s fine to use tutorials but you should know why you’re doing this and not something else. Maths hone your intuitive thinking as well. Given a new problem how to approach it from ground up Last but not the least Be good at Python. Don’t write crappy unreadable and non reproducible code ML is all about reproducibility of experiments.

304

10 Comments

LLM Visualization LLMs are advanced deep-learning models based on transformer decoders. It is essential to understanding various components of LLMs like - Embeddings - LayerNorm - Self Attention - Projection - MLP - Softmax - Output #llms #visualization #nlproc #deeplearning #embedding

1,191

35 Comments

📝 A Curated Set of NLP Primers: Attention, Autoregressive vs. Autoencoder Models, Transformers, BERT 🔹 Attention: http://attention.aman.ai - Overview (The Attention Mechanism, The Bottleneck Problem, The Context Vector Bottleneck, How Attention Solves the Bottleneck Problem) - Origins of Attention - Attention: Under the Hood - Classic Sequence-to-Sequence Model - Sequence-to-Sequence Model with Attention - Context Vector - Compared: Attention Mechanism v/s Fixed-length Context Vector Approach - Extensions to the Classic Attention Mechanism - Self-Attention (Why Have Multiple Attention Layers?) - Comparative Analysis: Additive vs. Scaled Dot-Product Attention (Computational Efficiency, Theoretical Complexity, Usage and Performance, Implementation Details) - Multi-head Attention - Cross Attention - Ghost Attention - Linear Attention 🔹 Autoregressive vs. Autoencoder Models: http://autoregenc.aman.ai - How do Decoder Models Work? - How do Encoder Models Work? - Encoder-Decoder Models - XLNet: the Best of Both Worlds (Permutation Language Modeling) 🔹 Transformers: http://transformer.aman.ai - Mathematical background (Vectors, Matrix Multiplication, Dot Product, Masking, Sampling) - Attention (Additive/Multiplicative Attention, Self/Cross-Attention, Multi-Head Attention) - Core components of the Transformer Architecture (Embeddings, Positional Encoding, Skip Connections, Layer Normalization, Softmax) - Top-level Transformer Architecture (Encoder and Decoder stack) - Implementation details (Byte-Pair Encoding, Teacher Forcing, Label Smoothing) - Lessons learned (What are Transformers learning? Why is training them so hard?) - Pros/cons of Transformers relative to CNNs/RNNs - Relation between Transformers and Graph Neural Networks 🔹 BERT: http://bert.aman.ai - Background: Pre-Training - Contextualized Embeddings - Masked Language Modeling (MLM) - Next Sentence Prediction (NSP) - BERT’s Encoder Architecture vs. Other Decoder Architectures - The Strength of Bidirectionality - Supervised Finetuning #artificialintelligence #deeplearning #nlp

66

4 Comments

AI/ML role descriptions for students and everyone else looking to switch, also firms trying to hire but still new in the field. —— 1) ML Engineers: Build the ML models and systems that solve various problems. Should understand ML algorithms and how to implement them at various scales, as well as design intelligent systems in production. Keywords - engineering, systems Education - CS/maths degree + training 2) Applied/Data Scientists: Use scientific principles to design solutions to real-world problems. Think ML for supply chain optimisation, ML for music recommendation. They are domain experts, ML applied to a particular domain and often forward findings to MLEs who build it out in production. Keywords - analysis, studies Education - maths/stats and related degrees 3) AI Engineers: Build end products by adapting existing ML models. Should be able to architect solutions for end users using existing models. Think using an LLM built by the MLE & DS to design AI Student Coach, AI Legal Advisor. Keywords - product, full stack Education - courses (uni can’t teach you this) 4) Data Engineers: Build pipelines for acquiring, processing and storing massive datasets for training these ML models. Keywords - datasets, systems Education - courses (you learn by practice) ——— 1) Research Scientists: Propose novel ML algorithms with rigorous theoretical roots. Very academic and ultra-premium math skills. Goes deep into an ML specialty. Keyword - maths, problem-solving Education - Strong Maths/ML/CS PhD, postdocs, professors. 2) Research Engineers: Build, train and benchmark novel ML algorithms at varying scales. The most versatile with broad ML coverage and difficult interview process at top labs. Research Scientists are specialists while Research Engineers are generalists. Keywords - Everything Education - ML Masters or PhD + 3y broad MLE experience 3) ML Software Engineer: Builds the infrastructure, platforms, libraries and tools that support these ML activities. Keywords - infra, cloud, HPC Education - CS (you could end up working on compilers or distributed systems which need academic CS knowledge) —— Maybe pursue paths you will enjoy and hire whom your company actually needs.

88

3 Comments

Releasing next weekend: Heterogeneous Graphs and Relational Graph Neural Networks (RGCNs): Part 12 of my Graph Series of Blogs This article will cover the following aspects of RGCNs: 1) What are Heterogeneous Graphs? 2) Knowledge Graphs and Heterogeneous Graphs 3) Simple Graph Neural Networks (GCNs) vs Relational Graph Neural Networks (RGCNs) 4) The overfitting problem in RGCNs 5) The Block Diagonal and Basis Matrices approach to address overfitting in RGCNs 6) Solving a Node Classification problem in Heterogeneous Graphs with RGCNs 7) Solving a Link Prediction problems in Heterogeneous Graphs with RGCNs Watch this space! #graphs #machinelearning #deeplearning

5

optimizing Large Language Models (LLMs) from a software engineering perspective can significantly impact latency and costs. with my recent time invested in building production-grade RAG solutions, here are a few key pointers I felt can be taken into consideration: - Long context prefill in the prompt passed to the LLM can increase latency and costs. opt for prompt compression or decomposition to mitigate this issue. - avoid using an agentic framework with multi-turn behavior, like self-iterating loops, or techniques like self-consistency prompting, to prevent higher latency and LLM costs. agentic frameworks like crewAI or autogen by ms can be overkill with higher inference time - if using vector embeddings, opt for a vector db with lower retrieval latency, such as sqlite-vss or redis. - implement semantic caching to reduce LLM API calls and lower response time. reducing output length can also aid in processing or decoding phases. you can also explore using YAML instead of JSON for structured output as YAML is less verbose and hence, fewer tokens, however, I still believe this is task-dependent. - tasks performed by LLMs that are not strictly sequential can be split into parallel calls. sequentially required steps can be merged into a single LLM call to avoid round-trip latency. - ensure proximity between LLMs and tools infrastructure-wise. Allowing third-party plugins to deploy on the infrastructure where large models run can further reduce latency. - using async programming paradigms ( for eg. concurrent API calls ) or parallel programming ( for eg. using multi-threading for parallel API requests ) can also help reduce the inference time and increase the throughput. do check LiteLLM (YC W23) though. - using query routing strategies like semantic router by Aurelio AI or Martian or Unify or RouteLLM by Anyscale and LMsys can also help boost cost efficiency and speed. I am yet to test these out though, but hopefully soon. some notes: - there is a tradeoff between throughput and time per output token: if we process let’s say multiple user queries concurrently, we'll have higher throughput compared to running the queries sequentially, but we'll take longer to generate output tokens for each user. - large language Models (LLMs) generate text in a two-step process: "prefill", where the tokens in the input prompt are processed in parallel, and "decoding", where text is generated one 'token' at a time in an autoregressive manner. Each generated token is appended to the input and fed back into the model to generate the next token. Generation stops when the LLM outputs a special stop token or when a user-defined condition is met (e.g., some maximum number of tokens has been generated). what are the other potential strategies for building production-grade rag solutions? #SoftwareEngineering #LLMOptimization #DataProcessing #GenAI #llms

42

10 Comments