Zhuojie Huang

- SwiftKV is a novel model transformation and distillation procedure (by folks at Snowflake) designed to optimize large language model (LLM) inference. Very cool findings in the paper (as presented in the vLLM meetup on 11/13). - SingleInputKV: Prefills later layers' KV cache using earlier layer outputs, reducing computation on prompt tokens. The key point is that the first few layers of attention in LLMs are similar (nice find), allowing SwiftKV's SingleInputKV mechanism to skip computations in later layers during inference. By using a single earlier layer's output to compute the KV cache for subsequent layers, it significantly reduces both computation (by 50%) and memory usage (by 62.5%) without sacrificing model performance. - AcrossKV: Merges KV caches of neighboring layers to minimize memory usage and increase batch size for higher throughput. - For models like Llama-3.1-8B and 70B, SwiftKV reduces prefill compute by 50%, memory usage by 62.5%, and achieves up to 2× higher throughput and 60% lower time per token. https://lnkd.in/guhEBW5G

57

Sharing my latest article on Overconfidence in State of the Art LLMs. Explore the challenges and limitations these models face in critical applications. #AI #MachineLearning #LLMs #ArtificialIntelligence #TechInsights #Innovation #EnterpriseSolutions #AIResearch

23

3 Comments

💡 Enhancing Retrieval Accuracy with Contextual RAG As GenAI systems go beyond demos and are deployed at production scale, improving retrieval accuracy remains a top priority for building robust Retrieval-Augmented Generation (RAG) systems. One promising technique that Anthropic recently discussed on their blog? Contextual RAG. The idea is to run LLMs on document chunks first to create context before running them through the standard embeddings pipeline. This method alone can reduce failure rates by 35%. And adding additional techniques like BM25 indexing and Reranking - techniques that are already well researched - improves failure rates by 67% 🤯 As token costs continue to drop and smaller models improve, these kinds of approaches become increasingly viable for large-scale use cases. You could, for example, use a smaller LLM for context creation while leveraging a larger, more powerful model for generating responses. Combining models in this way can help strike a balance between performance and cost. #AI #LLMs #RAG #ContextualAI #ScalableAI

25

3 Comments

The idea of agents has been around for a while. What’s new here? In his talk, Andrew Ng explains agentic systems that use design patterns such as reflection, tool use, planning, and multi-agent collaboration to outperform the results of zero-shot prompting. #llms #aiagent #ml #rlhf #mlops #datascience #llmops https://lnkd.in/d2Q8Yj7X

6

🚀 Fine-tuning Flux for Precision Image Generation! 🖼️ Recent work with Flux, the cutting-edge image generation model from Black Forest Labs, has revealed its immense potential. Flux represents a significant leap in AI-powered image creation, offering unparalleled quality and control. Key features of Flux: • Built on advanced flow-matching technology • Produces high-fidelity images with remarkable consistency • Offers faster inference times compared to traditional diffusion models 🔧 Fine-tuning Flux: Recent experiments with fine-tuning Flux using LoRA (Low-Rank Adaptation) have shown promising results. This technique allows for efficient customization of the model for specific styles or domains without the need for extensive computational resources. I've been fortunate to work on fine-tuning Flux for specialised applications and it is truly impressive. Have you worked with Flux or other AI image generation models? Share your thoughts on the future of AI in visual content creation! #AI #MachineLearning #ComputerVision #Flux #ImageGeneration

8

1 Comment

Nerd alert: Excellent article from Philip Rathle on improving GenAI and RAG with a graph DB. Key points: * Vector-based RAG increases response quality, but it does not provide certainty of a correct answer, and lack context and clear connection to confirmed truths * Knowledge graphs combined with RAG is a promising technique for improving LLM response quality to address these shortcomings (accuracy, explainability, traceability) * At a high level, this is done by including data represented as a graph as context to the LLM prompt https://lnkd.in/eHgRwmat

19

1 Comment

I'm excited to share two papers from the Retrieval team at Walmart that were presented at CIKM last week. They're both about improving the relevance of Embedding-Based Retrieval. 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲 𝗙𝗶𝗹𝘁𝗲𝗿𝗶𝗻𝗴 𝗳𝗼𝗿 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴-𝗯𝗮𝘀𝗲𝗱 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹. https://lnkd.in/gNQe7QhA The first is about how to intelligently choose a query-dependent cutoff on the cosine score to remove irrelevant items from the retrieved set. Since cosine scores aren't comparable across queries, we introduce a "Cosine Adapter" to map the scores to interpretable scores. 𝗘𝗻𝗵𝗮𝗻𝗰𝗶𝗻𝗴 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲 𝗼𝗳 𝗘𝗺𝗯𝗲𝗱𝗱𝗶𝗻𝗴-𝗯𝗮𝘀𝗲𝗱 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗮𝘁 𝗪𝗮𝗹𝗺𝗮𝗿𝘁. https://lnkd.in/gdqkc2hR The second is about model training techniques we learned to improve the relevance of embedding models. In particular, we describe how to use a teacher model to provide relevance labels for training Juexin (Joy) Lin Nicholas Rossi, PhD Feng LIU Zhen Yang Sachin Yadav Praveen Reddy Suram Satya Chembolu Prijith Chandran Hrushikesh Mohapatra Alessandro Magnani Ciya Liao

147

1 Comment

Text embeddings have become crucial for transforming textual data into dense vector representations (with context), enabling efficient processing across various modalities like text, images, and audio. This article provides a good summary on some of these (not all) embeddings available for text https://lnkd.in/gKp4ta9a 1. Gensim Word Embeddings (Free) Why Use It: Specializes in NLP with open-source models like Word2Vec and FastText, promoting transparency and ease of custom training. 2. Facebook Embeddings (Free) Why Use It: Trained on extensive corpora and supports over 100 languages, making it versatile for global applications. 3. AllenNLP Embeddings (Free) Why Use It: Provides NLP-focused embeddings like BERT and ELMo, with tools for fine-tuning and visualization. 4. OpenAI Embeddings (Paid) Why Use It: Comprehensive training on massive datasets allows for effective semantic understanding, making it suitable for advanced NLP tasks. Ideal for tasks requiring zero-shot learning. 5. MultiLingual BERT (Free) Why Use It: Handles multilingual datasets effectively, offering robust support for multiple languages. Perfect for applications requiring consistent performance across different languages, such as global customer support systems. 6. DistilBERT Base Uncased (Free) Why Use It: Smaller and faster than BERT, maintaining near-comparable performance with reduced resource usage. 7. MPNet V2 (Free) Why Use It: Excels in text similarity tasks with a Siamese architecture, providing accurate similarity measurements. 8. HuggingFace Embeddings (Free/Paid) Why Use It: Offers versatility with a wide range of embeddings (text, image, audio) and the ability to fine-tune on custom data. HF also provides wrappers to almost all open sourced models. 9. Longformer Base 4096 (Free) Why Use It: Supports long texts up to 4096 tokens, making it unique in handling lengthy documents without truncation. 10. RoBERTa (2022) (Free) Why Use It: Offers updated text representations with comprehensive training on recent data. 11. GTE-Base (Free) Why Use It: Offers a good balance of performance and resource requirements with a 768-dimensional embedding. 12. GTE-Large (Free) Why Use It: Provides high-quality embeddings with more detailed representations, useful for nuanced text analysis. Best for applications requiring in-depth text understanding, such as detailed content recommendations. 13. GTE-Small (Free) Why Use It: Optimized for speed with a smaller model size, making it efficient for applications needing quick processing. Useful for real-time systems where fast text similarity searches are critical. 14. E5-Small (Free) Why Use It: Combines compact size with good performance, making it suitable for fast, general-purpose text embedding tasks.

11

AI tip of the day: How do you know what LLM suits your GenAI project? 🤔 There is a tradeoff between LLM quality, speed (latency), and price - the LLM holy trinity. 😇 I used to struggle to compare models and synthesize this information until I came across Artificial Analysis. The site provides an independent analysis of various providers' AI models. 🚀 Check it out, it will save you a lot of time and money! 🙌

35

3 Comments

What happens when even AI starts overthinking? I recently had the opportunity to explore OpenAI’s latest #ChatGPT-o1 models (AKA ‘#StrawberryProject,’ 🍓) and observed something fascinating. The o1-preview model, despite its sophistication, fell into the trap of overcomplicating a task and ultimately failed to deliver the correct answer. In contrast, the o1-mini - a faster, more streamlined version - nailed it. The o1 models are trained using Reinforcement Learning, breaking down responses to verify assumptions along their chain of thought, ensuring the reasoning holds together. o1-mini is suggested by openAI as the faster and 80% (!) cheaper option. It is said to be particularly effective at coding, but as demonstrated here, also did a better job cracking this placement riddle. Sometimes, less is more. So, next time you’re stuck in an overthinking loop, maybe it’s time to switch gears. Could your "mini" moments be your best ones? 😉 #reinforcementLearning DuoTech ∞ #AI

19

2 Comments

Another insightful study from James Zou on the importance of continuous monitoring and assessment of Language Models (LLMs) to ensure their reliability and effectiveness across various applications. Zou and his team have shared their curated collection of questions and answers from GPT-3.5 and GPT-4, encouraging further research in this field. Check out the paper to learn more about the significance of LLMs and their impact on the future of natural language processing. #LLM #LanguageModels #NaturalLanguageProcessing #GPT3.5 #GPT4 https://lnkd.in/evy4V6FD

24

6 Comments

Can LLM preference tuning degrade Multi-Agent system performance? In a recent tweet, Andrej Karpathy questioned why all LLMs sound similar, prompting Sebastian Rascha to speculate about the uniformity of alignment datasets. Aligning user preferences is a last-mile step in training large language models. Regardless of the pre-training datasets used, could the preference datasets used during alignment compromise the effectiveness of LLMs? This issue could adversely affect multi-agent systems incorporating multiple LLMs in their design.

7

1 Comment

This is a big deal - geospatial algorithms such as DBSCAN, Getis-Ord have always been used to identify patterns in geospatial data. However, such algorithms have been limited to small workloads that can only fit on a single machine. Today, Wherobots provides a GeoStats toolbox that elastically scales up and down as you go based upon your data and workloads sizes. Try it out Today.

59

2 Comments

Graph databases are a game-changer in how we handle complex data relationships. Instead of rows and tables, you get nodes, edges, and properties—a structure that's built for speed and flexibility when dealing with connected data. Here’s a real-world example: In retail, we used graph databases to power our "Frequently Bought Together" recommendations. Every product is a node, and every co-purchase is an edge. This lets us quickly identify patterns—like how eyeliners in the beauty category are often bought with mascara. The result? Instant, data-driven product recommendations that boost cross-sells and enhance the shopping experience. On the tech side, this setup scales efficiently even with millions of transactions. Queries that would be heavy in a relational database—like finding all products frequently bought together—are lightning-fast with a graph. Graph databases aren’t just for social networks or fraud detection; they’re a tool we can leverage for smarter, more personalized retail experiences. If you’re dealing with interconnected data, it’s time to look into graph tech. #GraphDatabases #RetailTech #DataEngineering #AI #Personalization

17