Julen Arizaga Echebarria

Torch Compile is here to optimize inference in large language models (LLMs) with minimal overhead. In Benjamin Marie's latest article, explore how adding just one line of code can improve decoding performance. #Llama LLM

14

This paper (https://lnkd.in/g4QamVwg) describes the winning solution for the Amazon KDD Cup 2024, a Multi-task Online Shopping Challenge for LLMs. In this challenge, we introduce the ShopBench dataset, a collection of questions written in natural language covering various aspects of knowledge in online shopping: Shopping Concept Understanding, Shopping Knowledge Reasoning, User Behavior Alignment, and Multi-lingual Abilities. Here are the key points: Challenge: ► Build a helpful shopping assistant that can answer questions across various domains of online shopping. ► The competition provided a limited dataset (96 questions) and evaluated models on a hidden test dataset (ShopBench) with 20,000 questions. ► Participants submitted code and model weights, which were run on Amazon's infrastructure with limited compute resources and time constraints. Winning Solution (from Nvidia) [Slides-https://lnkd.in/gviMqx_m]: ► Fine-tuned LLM: The team used Qwen2-72B, a large language model, fine-tuned on a dataset they created by processing multiple public datasets and using other LLMs for data augmentation. ► Ensemble Learning: They employed multiple LoRA adapters, which are smaller models that can be added to a base LLM to improve performance on specific tasks. ► Addressing Distribution Shift: Wise-ft, a technique for adjusting the model's weights to account for the difference between training and test data, was used. ► Constraining Model Output: Logits processors were implemented to ensure the model generates outputs in the desired format (e.g., only digits for multiple-choice questions). ► Inference Optimization: The model was quantized to 4 bits and used with vLLM library to run efficiently within the time limits on the provided hardware (4x NVIDIA T4 GPUs). Other winning solutions here: https://lnkd.in/gnRZYSjx

6

LLMs have become more capable with better training and data. But they haven’t figured out how to “think” through problems at test-time. The latest research from Scale finds that simply scaling inference compute–meaning, giving models more time or attempts to solve a problem–is not effective because the attempts are not diverse enough from each other. 👉 Enter PlanSearch, a novel method for code generation that searches over high-level "plans" in natural language to encourage response diversity. PlanSearch enables the model to “think” through various strategies before generating code, making it more likely to solve the problem correctly. The Scale team tested PlanSearch on major coding benchmarks (HumanEval+, MBPP+, and LiveCodeBench) and found it consistently outperforms baselines, particularly in extended search scenarios. Overall performance improves by over 16% on LiveCodeBench from 60.6% to 77%. Here’s how it works: ✅ PlanSearch first generates high-level strategies, or "plans," in natural language before proceeding to code generation. ✅ These plans are then further broken down into structured observations and solution sketches, allowing for a wider exploration of possible solutions. This increases diversity, reducing the chance of the model recycling similar ideas. ✅ These plans are then combined before settling on the final idea and implementing the solution in code. Enabling LLMs to reason more deeply at inference time via search is one of the most exciting directions in AI right now. When PlanSearch is paired with filtering techniques—such as submitting only solutions that pass initial tests—we can get better results overall and achieve the top score of 77% with only 10 submission attempts. Big thanks to all collaborators on this paper including: Evan Wang, Hugh Zhang, Federico Cassano, Catherine Wu, Yunfeng Bai, William Song, Vaskar Nath, Ziwen H., Sean Hendryx, Summer Yue 👉 Read the full paper here: arxiv.org/abs/2409.03733

179

2 Comments

Combining the power of TensorFlow, NeuralForecast, and zero-shot LLMs, Jinhang Jiang shows how we can generate accurate multi-step quantile forecasts.

11

Explore the power of neural networks with PyTorch! Vipin P explains how the torch.nn module enables developers to define layers and configurations easily, allowing a focus on the overall architecture without getting distracted by weight and bias settings. 🔍 🔗 https://lnkd.in/evP7C36r #TheDeepHub #DeepInData #PyTorch #NeuralNetworks #MachineLearning

1

Here’s a blogpost that breaks down our work on Attribute Controlled Fine-tuning for Large Language Models . We introduced a new method that trains an auxiliary model to control a specific attribute (in this case toxicity). The approach regularizes the LLM's training by penalizing deviations from the desired (non-toxic) distribution, using the auxiliary model trained alongside the core LLM. This work, published at #EMNLP2024, was led by our intern Tao Meng (UCLA) together with our collaborators Ninareh Mehrabi, Palash Goyal, PhD, Anil Ramakrishna, PhD, Aram Galstyan, Richard Zemel, Kai-Wei Chang and Rahul Gupta. Blogpost: https://lnkd.in/eRzvKSKi Paper: https://lnkd.in/eHNwgWAt

34

Text embeddings have become crucial for transforming textual data into dense vector representations (with context), enabling efficient processing across various modalities like text, images, and audio. This article provides a good summary on some of these (not all) embeddings available for text https://lnkd.in/gKp4ta9a 1. Gensim Word Embeddings (Free) Why Use It: Specializes in NLP with open-source models like Word2Vec and FastText, promoting transparency and ease of custom training. 2. Facebook Embeddings (Free) Why Use It: Trained on extensive corpora and supports over 100 languages, making it versatile for global applications. 3. AllenNLP Embeddings (Free) Why Use It: Provides NLP-focused embeddings like BERT and ELMo, with tools for fine-tuning and visualization. 4. OpenAI Embeddings (Paid) Why Use It: Comprehensive training on massive datasets allows for effective semantic understanding, making it suitable for advanced NLP tasks. Ideal for tasks requiring zero-shot learning. 5. MultiLingual BERT (Free) Why Use It: Handles multilingual datasets effectively, offering robust support for multiple languages. Perfect for applications requiring consistent performance across different languages, such as global customer support systems. 6. DistilBERT Base Uncased (Free) Why Use It: Smaller and faster than BERT, maintaining near-comparable performance with reduced resource usage. 7. MPNet V2 (Free) Why Use It: Excels in text similarity tasks with a Siamese architecture, providing accurate similarity measurements. 8. HuggingFace Embeddings (Free/Paid) Why Use It: Offers versatility with a wide range of embeddings (text, image, audio) and the ability to fine-tune on custom data. HF also provides wrappers to almost all open sourced models. 9. Longformer Base 4096 (Free) Why Use It: Supports long texts up to 4096 tokens, making it unique in handling lengthy documents without truncation. 10. RoBERTa (2022) (Free) Why Use It: Offers updated text representations with comprehensive training on recent data. 11. GTE-Base (Free) Why Use It: Offers a good balance of performance and resource requirements with a 768-dimensional embedding. 12. GTE-Large (Free) Why Use It: Provides high-quality embeddings with more detailed representations, useful for nuanced text analysis. Best for applications requiring in-depth text understanding, such as detailed content recommendations. 13. GTE-Small (Free) Why Use It: Optimized for speed with a smaller model size, making it efficient for applications needing quick processing. Useful for real-time systems where fast text similarity searches are critical. 14. E5-Small (Free) Why Use It: Combines compact size with good performance, making it suitable for fast, general-purpose text embedding tasks.

11

Want to know how to build smarter, context-aware AI applications? In Ike Gabriel Yuson’s talk at PyConf Mini Davao 2024, he demonstrated how integrating LlamaIndex with Amazon Neptune and Amazon Bedrock powers Retrieval-Augmented Generation (RAG) for next-gen AI solutions. Attendees discovered how these tools enhance data retrieval, reasoning, and content generation. Catch the highlights! #GraphRAG #KnowledgeGraphs #LLM #DataScience #PyConfMiniDavao #Tech #Davao

5

1 Comment

TensorFlow vs. PyTorch: Finding the best Deep Learning framework for your AI journey 🚀 Which framework will boost your career goals? Whether you’re excited about diving into cutting-edge research with PyTorch or eager to create scalable AI models with TensorFlow, each option has its own perks! From how easy they are to use and the support you can get from the community to their flexibility and scalability, understanding these differences can really set you up for success in the fast-paced world of AI. I think Juliette Carreiro's article for @ironhack is a fantastic resource for anyone starting their journey in Data Science and Machine Learning! On another note, I’m still looking for a German-speaking Data Manager to join a leading company on an exciting automotive project in Frankfurt. If this sounds like a good fit for you or someone you know, please reach out here or at [email protected]. Happy to share more details! #DeepLearning #AI #DataScience #MachineLearning #TensorFlow #PyTorch #CareerInTech #Hiring #ironhack #redsofa #datamanager

12

1 Comment

𝗘𝘅𝗰𝗶𝘁𝗶𝗻𝗴 𝗡𝗲𝘄𝘀 𝘄𝗶𝘁𝗵𝗶𝗻 𝘁𝗵𝗲 𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗟𝗟𝗠𝘀) 𝗦𝗽𝗮𝗰𝗲! PyTorch has officially released TorchTune, a new tool designed to fine-tune Large Language Models (LLMs) with ease and efficiency. This groundbreaking tool promises to streamline the fine-tuning process, making it more accessible to researchers and practitioners alike. https://lnkd.in/eMf6dNZJ 𝗧𝗼𝗿𝗰𝗵𝘁𝘂𝗻𝗲 𝗢𝗳𝗳𝗲𝗿𝘀: ▪ 𝗣𝘆𝗧𝗼𝗿𝗰𝗵-𝗻𝗮𝘁𝗶𝘃𝗲 𝗟𝗟𝗠 𝗶𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻𝘀: Build on familiar ground with composable building blocks for popular LLMs. ️ ▪ 𝗠𝗲𝗺𝗼𝗿𝘆-𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 𝗳𝗶𝗻𝗲-𝘁𝘂𝗻𝗶𝗻𝗴 𝗿𝗲𝗰𝗶𝗽𝗲𝘀: Train on consumer-grade GPUs with 24GB VRAM – no beefy machines needed! ▪ 𝗘𝘅𝘁𝗲𝗻𝘀𝗶𝗯𝗶𝗹𝗶𝘁𝘆 𝗮𝗻𝗱 𝗲𝗮𝘀𝗲 𝗼𝗳 𝘂𝘀𝗲:  YAML configs and support for common dataset formats streamline your workflow. 𝗙𝗼𝗿 𝘁𝗵𝗼𝘀𝗲 𝗶𝗻𝘁𝗲𝗿𝗲𝘀𝘁𝗲𝗱 𝗶𝗻 𝗱𝗶𝘃𝗶𝗻𝗴 𝗱𝗲𝗲𝗽𝗲𝗿, 𝗵𝗲𝗿𝗲 𝗮𝗿𝗲 𝘀𝗼𝗺𝗲 𝗿𝗲𝘀𝗼𝘂𝗿𝗰𝗲𝘀: Code: https://lnkd.in/gEgETe5t Blog: https://lnkd.in/gP3mctA6 Tutorials: https://lnkd.in/gvh2eEsU This is a fantastic opportunity to explore and innovate, I'd be very interested in hearing your thoughts. ▪ 𝗪𝗵𝗮𝘁’𝘀 𝘆𝗼𝘂𝗿 𝘁𝗮𝗸𝗲 𝗼𝗻 𝘁𝗵𝗶𝘀?  ▪ 𝗛𝗼𝘄 𝗱𝗼 𝘆𝗼𝘂 𝘀𝗲𝗲 𝗧𝗼𝗿𝗰𝗵𝗧𝘂𝗻𝗲 𝗶𝗺𝗽𝗮𝗰𝘁𝗶𝗻𝗴 𝘆𝗼𝘂𝗿 𝗰𝘂𝗿𝗿𝗲𝗻𝘁 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀 𝗼𝗿 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵? ▪ 𝗛𝗼𝘄 𝗱𝗼 𝘆𝗼𝘂 𝘁𝗵𝗶𝗻𝗸 𝗧𝗼𝗿𝗰𝗵𝗧𝘂𝗻𝗲 𝘄𝗶𝗹𝗹 𝗶𝗺𝗽𝗮𝗰𝘁 𝘁𝗵𝗲 𝗳𝘂𝘁𝘂𝗿𝗲 𝗼𝗳 𝗺𝗮𝗰𝗵𝗶𝗻𝗲 𝗹𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗔𝗜 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁? #AI #PyTorch #TorchTune #LLMs #MachineLearning

24

1 Comment

Easily generate one billion embeddings in under a day... and with limited budget! How? With Ray Data from Anyscale and Pinecone! Cheng Su and Roy Miara present this can't-miss session during the #DataAISummit tomorrow. Catch it in-person or online.

43

2 Comments

The more I read the DSPy docs, the more I’m convinced that it is the way forward for LLM frameworks ✨ Being able to compile an LLM pipeline into robust low-level pipelines is such an incredible design + optimization for zero/few-shot/ensemble setups So, RTFM 🖖

21

The paper "Scalable MatMul-free Language Modeling" by Rui-Jie Zhu et al. introduces a method to eliminate matrix multiplication (MatMul) operations in large language models (LLMs), significantly reducing memory usage and computational cost. Their MatMul-free models perform comparably to state-of-the-art transformers, scaling effectively up to 2.7 billion parameters. They achieved memory reductions of up to 61% during training and over 10x during inference by using a custom FPGA hardware solution, which highlights the efficiency of lightweight operations for future language model accelerators. For further details, you can read the full paper https://lnkd.in/eZidqJuk

2

Learn how you can convert a PyTorch model, trained on the web, to a mobile-optimized format. Madhur Zanwar describes the process using a few lines of code. He also discusses his journey and how one approach didn't yield satisfactory results while trying to run it in a React Native app. #machinelearning #pytorch #computervision #edgeml #reactnative #objectdetection #yolov5

14

🚀 New podcast alert! Tune in to this episode where Kai Yang, VP of Product at LandingAI, joins the Augmented Ops Podcast to discuss the necessity of a data-centric AI approach and the value of building tools over solutions. This episode is a must-listen for industrial leaders, shop floor operators, and citizen developers interested in the future of frontline operations. Listen now: https://lnkd.in/gFXXQZJx #DataCloudSummit #ComputerVision

14

1 Comment

FlexAttention: Simplifying Attention in PyTorch _________________________________________________________________ FlexAttention is a new PyTorch API that combines the flexibility of PyTorch with the performance of optimized attention methods. It allows researchers to implement various attention mechanisms easily without needing to write custom kernels. Key Points: - Flexibility with Performance: -- Traditional attention methods offer high performance but limited flexibility. -- FlexAttention provides a flexible API that supports many attention variants with minimal effort. - How It Works: -- Standard Attention: Calculates attention scores using queries, keys, and values, normalizes them, and computes the weighted sum of values. -- FlexAttention: Allows for modifications to the attention scores through a flexible scoring function, enabling customization for different types of attention mechanisms. - Examples of Attention Variants: -- Relative Position Encoding: Adds positional information to attention scores based on relative positions. -- ALiBi Bias: Applies a bias to attention scores to account for position differences, enhancing model performance. -- Causal Mask: Masks future positions in the sequence to ensure the model only attends to previous positions, useful for autoregressive models. Handling Sparsity: -- BlockMask: Efficiently manages sparse attention masks, such as those used in causal attention, by creating masks that prevent unnecessary computations. Learn More: https://lnkd.in/g58pxB4A

8

Fine-tuning LLMs with Reinforcement Learning from Human Feedback This blog post explores how Reinforcement Learning from Human Feedback (RLHF) can be used to fine-tune Large Language Models (LLMs) to align with human preferences. It delves into the concept of Proximal Policy Optimization (PPO) and how it is used in this context. The key takeaway is that RLHF offers a powerful framework to train LLMs that can produce human-aligned text, while still maintaining creativity. This paves the way for more natural and nuanced language generation in the future of AI and natural language understanding. Read the article: https://lnkd.in/d9EGf579 #AI #machinelearning #goML

8