Rajesh Das’ Post

Building a robust GraphRAG System for a specific use case -Part Two- ⛁🕸️ In my previous post, we laid the groundwork by crafting a robust dataset of questions and their corresponding Cypher queries. part one : https://lnkd.in/ejaijvgq In the part two of the series, where we take it a step further and fine-tune a powerful Llama 3.1 model using QLoRA for efficient text-to-Cypher translation. Medium : https://lnkd.in/edfMHUuf My Web-book : https://lnkd.in/epCwrw5B #AI #Tech #RAG #Ollama #llama3 #Neo4j

In my latest blog post, I delve into the challenge of activation quantization in large language models (LLMs). Through benchmarking the Llama 3.1-8B-Instruct model across various batch sizes on an A100 PCIe 80GB GPU, I compare performance metrics between the vanilla model, weight-only quantized models, and those utilizing both weight and activation quantization via Neural Magic's LLMCompressor and vLLM inference engine. The findings shed light on architectural hurdles that make activation quantization a critical factor in optimizing LLM inference. Codebase: https://lnkd.in/g-V3FDyW Blog : https://lnkd.in/gduxWWxB #LLM #Genai #ML #DL #inference #optimization #generativeai #Llama3 #huggingface #MachineLearning #BitsandBytes #Quantization #llmcompressor

NVIDIA Introduces Hymba 1.5B: A Hybrid Small Language Model Outperforming Llama 3.2 and SmolLM v2 Large language models (LLMs) like GPT-4 and Llama-2 are powerful but require significant computational resources, making them impractical for smaller devices. Attention-based transformer models, in particular, have high memory demands and quadratic computational complexity, which limits their efficiency. State Space Models (SSMs), such as Mamba, offer an alternative with lower complexity, but their limited memory recall hampers performance on complex tasks. Read the full article: https://lnkd.in/eaNS7nRN Paper: https://lnkd.in/evvPy7NZ

1/ Nvidia has introduced a new large language model called Llama-3.1-Nemotron-70B-Instruct, which has been optimized to provide helpful answers to user queries. It combines different training methods such as regression and Bradley-Terry models. 2/ Nvidia used two self-generated datasets to create the training data: HelpSteer2 with over 20,000 scored prompt response pairs and HelpSteer2-Preference with comparisons between every two responses to the same prompt. The combination of both approaches produced the best results. 3/ In alignment benchmarks such as Arena Hard, AlpacaEval 2 LC and GPT-4-Turbo MT-Bench, Llama-3.1-Nemotron-70B-Instruct achieved first place in each case, outperforming top models such as GPT-4o and Claude 3.5 Sonnet. The model is available for free in HuggingChat or from Nvidia. 👇 Read more #GenerativeAI #Nvidia

When deploying Large Language Models (LLMs) like GPT or LLaMA, one of the most crucial considerations is GPU memory. A question I’ve come across often in technical discussions is: How much GPU memory do you really need to serve an LLM? The answer involves a bit of math, but more importantly, it reflects your understanding of scalability and hardware optimization. For instance, serving a 70B parameter model like LLaMA in 16-bit precision requires around 168GB of GPU memory—meaning a single A100 GPU with 80GB wouldn’t be enough. You’d need at least two to handle it efficiently. Here’s a handy formula to estimate GPU memory: M = P × 4B × Q × 1.2 The Formula to Estimate GPU Memory • M is the GPU memory in Gigabytes. • P is the number of parameters in the model. • 4B represents the 4 bytes used per parameter. • Q is the number of bits for loading the model (e.g., 16-bit or 32-bit). • 1.2 accounts for a 20% overhead. By mastering this calculation, you can avoid hardware bottlenecks and ensure smooth LLM deployments. It’s an essential skill for anyone working with AI models in production. #AI #GPU #LLM #MachineLearning #TechTips #AIEngineering #LLMDeployment #AIInsights

FastGen: Cutting GPU Memory Costs Without Compromising on LLM Quality Researchers from the University of Illinois Urbana-Champaign and Microsoft proposed FastGen, a highly effective technique to enhance the inference efficiency of LLMs without any loss in visible quality, using lightweight model profiling and adaptive key-value caching. FastGen evicts long-range contexts on attention heads by the KV cache construction in an adaptive manner. Moreover, it is deployed using lightweight attention profiling, which has been used to guide the construction of the adaptive KV cache without resource-intensive fine-tuning or re-training. FastGen is capable of reducing GPU memory usage with negligible generation quality loss. Quick read: https://lnkd.in/gXXc_igb Paper: https://lnkd.in/gztx-CZg #ai

🚀 How to Fine-Tune a Llama 2 Model with Limited VRAM Using QLoRA 🚀 Check it out on: https://lnkd.in/gRg8uzWd Fine-tuning Large Language Models (LLMs) can seem daunting—especially with memory constraints. Here's a practical breakdown of my recent experience fine-tuning Llama 2 (7B parameters) on a T4 GPU (16GB VRAM) using QLoRA. #Overview of LLM Fine-Tuning: Pretraining is foundational but requires enormous resources. Instruction Tuning using Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) makes models better at responding accurately to human prompts. #Why QLoRA? Traditional fine-tuning was too VRAM-intensive, so we used QLoRA to make this process efficient: 4-bit precision drastically reduces memory usage. Paired with PEFT (Parameter-Efficient Fine-Tuning), QLoRA minimizes computational load without sacrificing model performance. #Key Steps in the Fine-Tuning Process: 1.Environment Setup: Leveraged transformers, accelerate, peft, trl, and bitsandbytes. 2.Loading Dataset and Configurations: Chose a structured instruction dataset and configured it for Llama 2's 4-bit quantization. 3.Loading the Model: Loaded the Llama 2 base model and tokenizer in 4-bit precision. 4.Configuring LoRA Parameters: Set parameters like LoRA rank and scaling for fine-tuning efficiency. 5.Training Setup: Defined training arguments with custom batch size, learning rate, and gradient checkpointing for stable training. 6.Running the Training with SFTTrainer: Supervised fine-tuning over 1,000 samples while logging results in TensorBoard for monitoring. 7.Saving and Testing the Model: Saved the new fine-tuned model and checked TensorBoard logs for performance insights. #Why This Matters: This approach shows that with the right tools, fine-tuning powerful models on limited hardware is achievable. We’re pushing boundaries by adapting models like Llama 2 to specialized tasks while keeping resources optimized. #Results: Our Llama 2 model is now ready to deploy on smaller devices, thanks to efficient memory management using QLoRA and PEFT techniques. This opens the door to deploying large models more effectively across various platforms. ➡️ Key Takeaway: Fine-tuning massive LLMs with QLoRA makes it possible to scale AI even with constrained hardware. #AI #MachineLearning #LLM #FineTuning #Llama2 #QLoRA #DeepLearning #NLP #ArtificialIntelligence #GPT #ParameterEfficient #DataScience #SupervisedLearning #RLHF #LoRA #ModelOptimization #TechInnovation #HuggingFace #DataDriven #FutureOfAI

1/ Nvidia releases Nemotron-4 340B, a free pipeline that generates high-quality synthetic data for training and tuning large-scale language models (LLMs). It can be used for commercial applications. 2/ The Nemotron-4 340B family consists of a base model trained on 9 trillion tokens, an instruction model for generating diverse synthetic data, and a reward model for filtering high-quality responses. 3/ In benchmarks, the instruction model typically outperforms other open-source and -weights models, and in some cases outperforms GPT-4. Nvidia also makes the models available for commercial use under an open model license.

Mirror, mirror on the wall, which is the fairest LLM fine-tuning framework of them all? In a new article on our blog, we break down the pros and cons of three of the most popular LLM fine-tuning libraries: Axolotl from Wing Lian, Unsloth AI from Daniel Han, Torchtune from Rafi Ayub and Rohan Varma at AI at Meta Takeaways: If you are a beginner: Use Axolotl. If you have limited GPU resources: Use Unsloth. If you prefer working directly with PyTorch: Use Torchtune. If you want to train on more than one GPU: Use Axolotl. https://lnkd.in/epYqNufp #ai #finetuning #llms #chatgpt #pytorch

NVIDIA introduces NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks, rivaling the leading proprietary models (e.g., GPT-4o) and open-access models (e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved text-only performance over its LLM backbone after multimodal training.