Manjunath Bhat

“The SVD is absolutely a high point of linear algebra” Professor Gilbert Strang Here I used SVD to compress my image but what else I can do with SVD well dimensionality reduction and noise reduction(which I'm not a big fan of) etc. Here I used a image of height and width 608 x 608 respectively with 3 color channels I calculated Left and right singular matrix for every color channel then selected k singular values that give the principal axis with most variance. I originally had 608 singular values but now I'm describing the same image with only 25 well 70-80 singular values has worked the best for my case as less data was lost and image was much clearer but I chose to stick with 25. What else it can be used for? Face Detection or recognition but we have better CNN models that work the best the accuracy with SVD is quite less. How I see SVD? Well, I have to use corporate jargon now I see SVD as describing the row and column space of a matrix by the U and V matrix that we get after taking left and right transpose of same matrix. How people see SVD? They see SVD as finding a vector in input space that is orthogonal and stays orthogonal after transformation in output space. The image size also reduced by half. #svd #linearalgebra #machinelearning

12

Euler's Totient Function Euler’s Totient function Φ (n) for an input n is the count of numbers in {1, 2, 3, …, n-1} that are relatively prime to n, i.e., the numbers whose GCD (Greatest Common Divisor) with n is 1 . Thanks to Sandeep Jain for this informative session on Euler Totient Function in Mathematics for Competitive Programming and all GeeksforGeeks Team . #softwareengineer #dsa #interviews #maths #sde #programming #c++ #linkedin #rahulverma

11

1 Comment

Training LLMs using reinforcement learning using human feedback. I was slacking on this topic for 6 months and finally was able to go through it, and wow I am amazed at what technologies are being worked on in production-ready systems these days. Generally in RL of LLMs, we have a reward model (let's say a good/bad text classifier) that gives a reward (prediction 1 for good, -1 for bad) and an RL algorithm (based on multiple loss functions) updates the weights of the LLM model to achieve better results. This is an iterative process between the reward model, the LLM, and the RL algorithm. Overall, my take is the most important aspect of training LLM using RLHM is obtaining that human feedback and training a reward model.

28

🚨ACL 2024 Outstanding Paper Award Alert 🚨 ➡️Paper Title: INDICLLMSUITE: A BLUEPRINT FOR CREATING PRE-TRAINING AND FINE-TUNING DATASETS FOR INDIAN LANGUAGES 🌟Few pointers from the paper 🎯Despite the considerable advancements in English LLMs, the progress in building comparable models for other languages has been hindered due to the scarcity of tailored resources. 👁In this paper Authors work aims to bridge this divide by introducing an expansive suite of resources specifically designed for the development of Indic LLMs, covering 22 languages, containing a total of 251B tokens and 74.8M instruction-response pairs. 🎯Recognizing the importance of both data quality and quantity, their approach combines highly curated manually verified data, unverified yet valuable data, and synthetic data. 🎯They built a clean, open-source pipeline for curating pre-training data from diverse sources, including websites, PDFs, and videos, incorporating best practices for crawling, cleaning, flagging, and deduplication. 🎯For instruction-fine tuning, they amalgamate existing Indic datasets, translate/transliterate English datasets into Indian languages, and utilize LLaMa2 and Mixtral models to create conversations grounded in articles from Indian Wikipedia and Wikihow. 🎯Additionally, they addressed toxicity alignment by generating toxic prompts for multiple scenarios and then generated non-toxic responses by feeding these toxic prompts to an aligned LLaMa2 model. 🎯 They hope that the datasets, tools, and resources released as a part of this work will not only propel the research and development of Indic LLMs but also establish an open-source blueprint for extending such efforts to other languages. 🏢Organization: 🛎️Nilekani Centre at AI4Bhārat🛎️ Indian Institute of Technology, Madras 🛎️Microsoft🛎️Sant Longowal Institute of Engineering and Technology 🛎️Indian Institute of Information Technology Design & Manufacturing Kancheepuram, 🛎️Maharaja Agrasen Institute Of Technology, Delhi 🛎️Sarvam ,🛎️ NICT 🧙Paper Authors: Mohammed Safi Ur Rahman Khan, Priyam Mehta , Ananth Sankar , Umashankar Kumaravelan, Sumanth Doddapaneni, Suriyaprasaad G, Varun Balan, Sparsh Jain, Anoop Kunchukuttan, Pratyush Kumar, Raj Dabre, Mitesh Khapra 1️⃣Read the Full Paper here: https://lnkd.in/gqwQST-r 2️⃣Code: https://lnkd.in/giWPVntx 🎉 🎉  Congratulations to all the authors 🎉 🎉 Find this Valuable 💎 ? ♻️REPOST and teach your network something new Follow me 👣, Naveen Manwani, for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements. #ACL2024 #Indicllms #llms #llama2 #mixtral

16

Happy to share that our paper, "UniMTS: Unified Pre-training for Motion Time Series", has been accepted to the Neural Information Processing Systems (NeurIPS) 2024. This is the first pre-trained model specifically designed for human motion time series, excelling in zero-shot activity recognition across a wide range of wearable devices, regardless of body placement or orientation. Our approach follows a CLIP-inspired pre-training framework, where we leverage contrastive learning to align motion data with LLM-enhanced activity descriptions. https://lnkd.in/gjJcVSiR #NeurIPS2024 #neurips #neurips2024 #neurips24 #MachineLearning

92

9 Comments

❝ How do you taste your Wine? With the right Glass, served from the right Bottle. How do you utilize your LLM? With the right Task, served from the right Compute environment. ❞ If the quote above caught your interest, check out my blog post that was featured on the Toyota Connected India Medium channel last week: Leveraging LLMs in Cloud - 🔗 https://lnkd.in/gfHsdevH --- I hope this blog adds a bit of amusement for the seasoned AI/ML engineers 🧠👨‍💻 with the carefully crafted images and perhaps even the silly wine analogies 😉. Whether you're a cloud engineer ☁️ 👨‍💻, a product owner of cloud-based apps, or someone new to Generative AI in the cloud, this blog offers a detailed, beginner-friendly look into the selection and deployment of LLMs in cloud environments. --- A heartfelt thanks 🙏 to my colleagues at Toyota Connected India for taking the time to review, shape this piece and even share a creative writeup like below in the original post 🫡 👇 🤔 Curious to dive in? Click the link below to explore the blog on Medium! 🌐 🔗 https://lnkd.in/gfHsdevH

18

Dealing with Class Imbalance in Multi-Class Classification Data-Level Strategies 1. Oversampling Minority Classes: • Use methods like SMOTE or ADASYN to generate synthetic samples for underrepresented classes. • Duplication of existing samples is simpler but can lead to overfitting. 2. Undersampling Majority Classes: • Remove samples from overrepresented classes to balance the dataset, though it risks losing important data. 3. Data Augmentation: • Apply transformations (e.g., rotations, flips) to create more diverse samples for minority classes, especially in image data. 4. Cluster-Based Sampling: • Use clustering to ensure diverse and balanced subsets during sampling. Algorithm-Level Strategies 1. Class Weights: • Assign higher weights to minority classes in the loss function to penalize misclassifications more heavily. 2. Cost-Sensitive Learning: • Adapt algorithms to consider higher misclassification costs for minority classes. 3. Ensemble Methods: • Bagging: Train base learners on balanced subsets of data. • Boosting: Algorithms like XGBoost can incorporate class weights or resampling. 4. One-vs-All (OvA) or One-vs-One (OvO): • Break the multi-class problem into binary classification tasks, which handle imbalance better. Hybrid Strategies 1. Combine Oversampling and Undersampling: • Balance datasets using a mix of these techniques while avoiding overfitting or excessive data loss. 2. GANs for Synthetic Data: • Use Generative Adversarial Networks to create realistic samples for minority classes. Evaluation Strategies 1. Balanced Metrics: • Use F1-score, precision, recall, and macro-averaged or weighted metrics instead of accuracy. • Analyze confusion matrices and ROC-AUC curves for all classes. 2. Stratified Cross-Validation: • Ensure class proportions are consistent across folds during training and validation. 3. Threshold Tuning: • Adjust class-specific decision thresholds to optimize performance metrics. Advanced Techniques 1. Transfer Learning: • Pretrain on a balanced dataset, then fine-tune on the imbalanced one. 2. Meta-Learning: • Use algorithms designed to adapt dynamically to imbalanced scenarios. 3. Bayesian Hyperparameter Optimization: • Fine-tune hyperparameters specifically for imbalanced data. Summary By combining data-level (e.g., oversampling, undersampling), algorithm-level (e.g., class weights, boosting), and hybrid strategies (e.g., GANs), and using robust evaluation metrics, you can effectively address class imbalance in multi-class classification tasks.

114

4 Comments

Sharing update from SSRN publishing... "Dear Ashutosh Bhatnagar: Your paper, "NEIGHBORHOODS OF PRIMES", was recently listed on SSRN's Top Ten download list for: Discrete Mathematics & Combinatorics eJournal and Number Theory eJournal. As of 02 October 2024, your paper has been downloaded 16 times. You may view the abstract and download statistics at: https://lnkd.in/eMmhurr4. Top Ten Lists are updated on a daily basis. Click the following link(s) to view the Top Ten list for: Discrete Mathematics & Combinatorics eJournal Top Ten and Number Theory eJournal Top Ten. Click the following link(s) to view all the papers in: Discrete Mathematics & Combinatorics eJournal All Papers and Number Theory eJournal All Papers. To view SSRN's Top Ten lists for any network, subnetwork, eJournal or topic on the Browse list (reachable through the following link: https://lnkd.in/eEmS8twm), click the "i" button to the right of the name, and then select the "Top Downloaded" link in the popup window. Your paper may be included in future Top Ten lists for other networks or eJournals. If so, you will receive additional notices at that time. Questions? Visit the SSRN Support Center. This email helps you gauge the impact of your paper. To unsubscribe from ALL notifications, including SSRN eJournals, click here. Thank you, The SSRN Team"

10

3 Comments

🔸Here's Todays Leetcode Problem of the day : [ 🔥DAY 193 ] 🔸Name : Find if Path Exists in Graph 🔸Description : There is a bi-directional graph with n vertices, where each vertex is labeled from 0 to n - 1 (inclusive). The edges in the graph are represented as a 2D integer array edges, where each edges[i] = [ui, vi] denotes a bi-directional edge between vertex ui and vertex vi. Every vertex pair is connected by at most one edge, and no vertex has an edge to itself. You want to determine if there is a valid path that exists from vertex source to vertex destination. Given edges and the integers n, source, and destination, return true if there is a valid path from source to destination, or false otherwise.   Example 1: YInput: n = 3, edges = [[0,1],[1,2],[2,0]], source = 0, destination = 2 Output: true Explanation: There are two paths from vertex 0 to vertex 2: - 0 → 1 → 2 - 0 → 2 Example 2: oInput: n = 6, edges = [[0,1],[0,2],[3,5],[5,4],[4,3]], source = 0, destination = 5 Output: false Explanation: There is no path from vertex 0 to vertex 5.   Constraints: 1 <= n <= 2 * 105 0 <= edges.length <= 2 * 105 edges[i].length == 2 0 <= ui, vi <= n - 1 ui != vi 0 <= source, destination <= n - 1 There are no duplicate edges. There are no self edges. 🔸Link: https://lnkd.in/dd3Wj4YZ

6

F1-Score vs Accuracy: Why F1-Score Should Be Your Focus. In machine learning, it's tempting to focus on ACCURACY—the percentage of correctly predicted instances. But what if I told you that accuracy isn't always the best indicator of your model's performance? In cases where the data is imbalanced (e.g., fraud detection, disease diagnosis), accuracy can be misleading. Imagine a scenario where only 5% of transactions are fraudulent—if your model predicts "non-fraudulent" 100% of the time, you’ll get 95% accuracy but miss all the critical fraud cases! When it comes to the F1-score: - Precision tells you how many of your positive predictions were correct. - Recall tells you how many of the actual positives you correctly identified. The F1-score balances both precision and recall, giving you a better sense of how well your model performs—especially when false positives and false negatives matter. If you're working on models where missing a positive instance could have serious consequences (like healthcare, security, or finance), the F1-score should be your go-to metric. Focus on the right metrics to make smarter models and better decisions. #MachineLearning #DataScience #AI #F1Score #ModelEvaluation #TechInsights #DataDriven #BrainK

10

Hey people! Remember the awesome work on alpha geometry from Google DeepMind It is impressive!! It solved international maths Olympiad probs !! But we extend it a bit so now we can solve even more math problems, including those from national math Olympiads!! A very short note on that - we don’t want to waste your time, you can manage to read a 2 pagers 📟😆😆😆 https://lnkd.in/eRq9mK-H #ai #MachineLearning

43

ANPD: Exciting advancements in LLM efficiency Researchers have developed a novel approach called Adaptive N-gram Parallel Decoding (ANPD) to address the resource consumption and latency challenges faced by LLMs. ANPD introduces a two-stage process: 1. Rapid Drafting: An adaptive N-gram module generates multiple tokens simultaneously, significantly speeding up the initial generation process. 2. Verification Phase: The original LLM meticulously evaluates and confirms the proposed tokens, ensuring the output's accuracy and fidelity. This innovative method retains the LLM's original quality while achieving impressive speed improvements. In experiments, models like LLaMA witnessed up to a 3.67x speedup! Key benefits of ANPD: * Enhanced Efficiency: Reduces resource consumption and latency. * Lossless Approach: Maintains the original LLM's output quality. * Plug-and-Play Enhancement: No retraining or additional GPU memory needed. ANPD represents a significant step forward in optimizing LLM performance, paving the way for wider adoption and more efficient applications. Read more about the Adaptive N-gram Parallel Decoding (ANPD) in this research paper: https://lnkd.in/gNzt2TK3 #LanguageModels #NaturalLanguageProcessing #AIResearch #MachineLearning #DeepLearning #Interpretability #ConceptBasedExplanations #CAVs #ModelUnderstanding #LLM #AI #Reasoning #NLP #MachineLearning #SelfPlay #AdversarialLearning #LLMs #AI #MachineLearning #NaturalLanguageProcessing #SelfLearning #Reasoning #Planning #scalinglaws #research #reproducibility #Breakthrough #ANPD #LanguageModels #Efficiency #Innovation

30

1 Comment

- A number theory problem - Find all positive integers “n” with the property s(n) = 3s(n+3) where s(n) stands for sum of digits of “n”. This problem can be specifically asked for “two/three/four digit numbers” in math contests”.

4

India's AI Opportunity: Building Applications on Top of Large Language Models 🇮🇳🚀 In an exclusive interview with The The Economic Times, Andrew Ng, a prominent figure in the AI world, shared his insights on India's potential in the AI landscape. Here are the key takeaways: • India's biggest AI opportunity lies in building applications on top of large language models (LLMs) and foundation models. • While training LLMs is capital-intensive, the real value lies in the application layer, which is more accessible to startups and enterprises. • India should focus on sectors where its economy is already strong, such as IT services, manufacturing, agriculture, and finance, to maintain its advantages through AI applications. • The generative AI moment has leveled the playing field for startups and enterprises, with India standing at the verge of opportunities. • Ng's AI Fund invests in startups building AI applications, with an average of one investment per month. They have also invested in an Indian healthcare AI startup, Jivi Health. • Ng believes that with the rise of AI, learning to code will become a significant personal productivity booster, similar to language literacy. #AI #MachineLearning #DeepLearning #IndiaTech #Startups #TechTrends

1

Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters Exploiting activation sparsity helps accelerate inference process of large language models (LLMs) without compromising performance. However, activation sparsity is determined by activation functions, and commonly used ones like SwiGLU and GeGLU exhibit limited sparsity. To address these challenges, this paper proposed a novel dReLU function, which is designed to improve LLM activation sparsity, along with a high-quality training data mixture ratio to facilitate effective sparsification. 𝗞𝗲𝘆 𝗰𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻𝘀: - Efficient dReLU activation function: proposed method utilizes fewer than 150B tokens, representing less than 1% of typical pretraining tokens (commonly 15T tokens). - Sparse activated models: released sparsely-activated TurboSparse-Mistral7B and TurboSparse-Mixtral-47B models, which shows better performance compared to original versions - Practical inference speedup: Evaluation shows that with released models, 2-5× speedup can be achieved, with up to 10 tokens/s even without a GPU on TurboSparse-Mixtral-47B. 𝗔𝗻𝗮𝗹𝘆𝘀𝗶𝘀 i) Limitations of existing ReLUfication - sparsity evaluation of ReLULlama-7B and original Llama-2-7B, shows that existing ReLUfication methods can only improve the sparsity from 40% to 67% - This can be attributed to the ReLU activation function applied after gate weight, which masks all negative activations to zero ii) dReLU - newly proposed activation function, where ReLU is applied after both the up- and gate-projection - for an experiment comparing models using dReLU and SwiGLU, results shows that dReLU structure exhibit similar convergence compared to those using SwiGLU -further evaluation results for various sparsity levels demonstrate dReLU activation function enables high sparsity without significant degradation in performance, maintaining competitive performance even at 90% sparsity - In contrast, the SwiGLU-based model experiences a severe increase in perplexity as sparsity reaches 80%. 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝗮𝗹 𝗥𝗲𝘀𝘂𝗹𝘁𝘀 - TurboSparse-Mistral-7B outperforms Gemma-2B by far, while only activating 3B parameters. - TurboSparse-Mixtral-47B outperforms the original Mixtral-47B with only 4.5B parameters activated - for activations with a value of zero, TurboSparse-Mistral-7B, on average, has 90% of the neurons inactive in each layer, and TurboSparse-Mixtral-47B with 85% on average for each expert FFN 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝗦𝗽𝗲𝗲𝗱𝘂𝗽 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 - On Pure CPU Inference, ReLUfied models achieve 2.08-2.28× speedup over original model. - On Hybrid GPU-CPU Inference, demonstrate significant speedups ranging from 2.52 to 4.64× compared to the baseline llama.cpp - For Deploying LLMs on mobile phones, achieves a 22.2× speedup using TurboSparse-Mixtral-47B inference compared to llama.cpp with original Mixtral-47B 𝗣𝗮𝗽𝗲𝗿: https://lnkd.in/eeHCWhUt 𝗠𝗼𝗱𝗲𝗹𝘀: https://lnkd.in/etHpZUju

24

1 Comment

Gerhann Pieters brought up this very interesting discussion on ECL modeling. IMO, provisions analysts should always worry about the final impact of parameters models on the ECL. This is basically how their models are proven feasible. But... there is a world of derived analysis that can rise from this - with respect to impacts in both the PnL and the business strategy. That's why major banks set provision steering committees to collect insights and different perspectives.

23

Is GPT-4o good at table recognition? Not quite! OCR looks fine, but tripped by spanning cell table structure! 🤩 However, our UniTable, with only 124M param trained by self-supervision, shows great understanding! 👉 UniTable paper: arxiv.org/abs/2403.04822 👉 UniTable code: https://lnkd.in/e4a7YMxJ #MachineLearning #GPT-4o #Table #Document

85

8 Comments

Exciting work by Mostafa Elhoushi and Akshat Shrivastava from Meta, who co-authored the paper "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding". In their work, a single model is used to leverage dynamic compute during generation to enable low-latency high quality predictions within a single model by using the earlier layers to generate tokens and verify with the remaining ones. Their algorithm is capable of taking advantage of self-speculation and producing high quality outputs at a lower latency while using a single model and KV cache. Their algorithm can be fine-tuned on top of any LLMs. Here's a link to the paper :-https://lnkd.in/dAJESKEF Thanks Mostafa Elhoushi and @Akshat Shrivastava for presenting. Thanks Daniel Lenton, Unify for this session.

36

2 Comments

📢 Exciting Research Alert! 🚀 📚 Just came across a groundbreaking paper titled "Linearizing Large Language Models" by Jean Mercat, Kushal Arora, Achal Dave, Igor Vasiljevic, Adrien Gaidon, Sedrick Keh, and Thomas Kollar from Toyota Research Institute. This paper introduces a novel approach called Scalable UPtraining for Recurrent Attention (SUPRA) to convert pre-trained language models into efficient Recurrent Neural Networks (RNNs). Let's dive into the key research points! 🔬 🔹 The paper addresses the challenge of inference cost in Transformers, which scales linearly with the number of tokens. To overcome this, the authors propose a linearization technique that converts pre-trained Transformers into RNNs, reducing the cost of inference. This technique leverages the strong pre-training data and performance of existing Transformer models while requiring only 5% of the training cost. 🔹 The authors introduce the concept of linear attention, which replaces the softmax dot-product attention with a more generic similarity function. They explore different similarity functions, including a linear kernel, and demonstrate the computational benefits of linear attention. This allows the models to be trained with sequence parallelism (as Transformers) but operate as RNNs at inference time. 🔹 The paper discusses the limitations of recurrent LLMs (Linear Language Models) compared to Transformers, particularly in in-context learning and long-context tasks. The authors compare pre-trained and uptrained RNNs to Transformers, revealing a persistent gap in performance for these tasks. 🔹 The methodology section provides a detailed explanation of linear attention, recurrent inference, and the process of finetuning a Transformer into an RNN. The authors build upon previous work and propose modifications to address stability issues and improve performance at larger scales. 🔹 The paper also highlights the importance of using modern relative positional encoding schemes, such as RoPE, for competitive performance in linear attention models. 🔗 For those interested in exploring the code and models, the authors have provided a GitHub repository: [GitHub Link](https://lnkd.in/eAivZekb). This repository contains the code and models related to the SUPRA approach, allowing researchers to replicate and build upon the findings of the paper. 📝 In summary, "Linearizing Large Language Models" presents a cost-effective alternative to pre-training linear Transformers by uptraining existing models into efficient RNNs. The paper explores the mathematical aspects of linear attention, recurrent inference, and the SUPRA methodology. This research opens up new possibilities for reducing inference cost and improving performance in language and multimodal models. Don't miss out on this groundbreaking work! 🌟 #ResearchPaper #LanguageModels #Transformers #RNNs #SUPRA #MachineLearning #NLP #ArtificialIntelligence #GitHub

2