Burak Emir

💥💥💥 Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs Tianyu Guo, Druv Pai, Yu Bai, Jiantao Jiao, Michael I. Jordan, Song Mei Abstract Practitioners have consistently observed three puzzling phenomena in transformer-based large language models (LLMs): attention sinks, value-state drains, and residual-state peaks, collectively referred to as extreme-token phenomena. These phenomena are characterized by certain so-called "sink tokens" receiving disproportionately high attention weights, exhibiting significantly smaller value states, and having much larger residual-state norms than those of other tokens. These extreme tokens give rise to various challenges in LLM inference, quantization, and interpretability. We elucidate the mechanisms behind extreme-token phenomena. First, we show that these phenomena arise in very simple architectures -- transformers with one to three layers -- trained on a toy model, the Bigram-Backcopy (BB) task. In this setting, we identify an active-dormant mechanism, where attention heads become sinks for specific input domains while remaining non-sinks for others. Our theoretical analysis of the training dynamics reveals that these phenomena are driven by a mutual reinforcement mechanism. Building on these insights, we propose strategies to mitigate extreme-token phenomena during pretraining, including replacing softmax with ReLU and Adam with SGD. Next, we extend our analysis to pretrained LLMs, including Llama and OLMo, showing that many attention heads exhibit a similar active-dormant mechanism as in the BB task, and that the mutual reinforcement mechanism also governs the emergence of extreme-token phenomena during LLM pretraining. Our results reveal that many of the static and dynamic properties of extreme-token phenomena predicted by the BB task align with observations in pretrained LLMs. 👉 https://lnkd.in/d-3mWsh9 #machinelearning

6

Interesting article covered in the New York Times that describes experiments by Anthropic to understand what's really going on inside of neural network-based Large Language Models (LLMs). Researchers identified clusters of millions of neurons related to "concepts." Although neural networks are "inspired by the human brain" and are not an actual brain, the ability to excise these clusters to remove a behavior or to stimulate/focus on them to emphasize a concept does sound a lot like brain surgery. The ability to identify such clusters does kind of sound like Functional MRI. It will be interesting to see whether research into a computer model that generates language teaches us something about our own biology. Because of course we are totally not just computer code running on someone's computer. https://lnkd.in/eEM_fEzX

8

2 Kommentare

🔍 Quantum AI Series: Modeling Sequences with Quantum States Picking up where I left off in my #QuantumAI series, let’s talk about Modeling Sequences with Quantum States, with insights from a fascinating paper on the topic. Classical probability distributions can be modeled using quantum states—particularly entangled states that retain information from subsystems. This approach allows quantum models to efficiently represent sequences without losing essential details. Tensor networks play a key role in optimizing the process, providing a compact, low-rank factorization of data. As I've shared previously, Singular Value Decomposition (SVD) is foundational to these models, as it helps represent the complex correlations present in sequential data. Matrix Product States (MPS) capture these rich interactions by using reduced densities to efficiently compress sequence representations while preserving their structure. This is the essence of keeping representation while building efficient lightweight models. In summary, quantum probability models allow us to go beyond classical limits in sequential tasks, contributing to information retention. Sharing the paper for a deeper dive if you'd like so: https://lnkd.in/d5aqXyyS #QuantumAI #TensorNetworks #SVD #QuantumComputing #AI

5

Kudos to #Composable_kernel team and please check out the AMD engineer Haocong Wang presents the ROCm Composable Kernel library which allows to write high-performance kernels for machine learning with HIP C++. #Composable_kernel #ROCm

39

1 Kommentar

🌍 Live from Berlin are thoughts on eBPF, Linux Kernel, Tracing, Auto-Discovery, Anomaly Detection, Edge Delta... Awesome technical talk by our very own Ozan Sazak as he walks through writing and implementing eBPF within an application at WeAreDevelopers conference. 💯 What makes it so powerful? Engineers cannot predict the future, and within #observability it's very difficult to find the right logic and query to ensure you have coverage for tomorrow's production outage. eBPF gives a reliable and standardized stream of data that allows you to have a lot more coverage than simply relying on logs and metrics. If you're curious about what we're doing in this space, come talk to us.

62

3 Kommentare

Mustafa Suleyman explains that we need to be able to describe AI and that the names, mental models, and metaphors matter. I agree. We need to describe what AI is if we are to understand it, which often means moving beyond names, metaphors, and mental models. However, he introduces even more opaque metaphors like "digital species" and says we need to be able to explain AI to 6-year-olds, though he never says why. Explaining AI to a 6-year-old doesn't advance the adoption, regulation, or innovation discussion. In fact, it would harm such discussions. Simplifications suitable for a first-grader hardly scratch the surface of the complex ethical, technical, and regulatory challenges ahead. We need not just awareness at the playground level but deep, informed discussions among policymakers and technologists. Children are our future, and we should consider them when having these discussions, but the discussions themselves don't need to be accessible to a child. Telling a child that AI is like a "digital species" doesn't help legislators decide on privacy regulations or guide researchers in avoiding bias. If the future of AI depends on the understanding of a 6-year-old, we might as well let kindergarteners vote on data privacy laws. Let's hope Mustafa doesn't start handing out juice boxes and nap time schedules at the next big AI conference! Though this kind of content is on trend for Ted talks.

5

3 Kommentare

This is a great first step in understanding how scaling dynamics extend beyond language to other types of sequential data. Time is a universal axis, understanding how models learn optimally over data collected over time is a crucial step in the broader AI research agenda.

26

Very insightful work by Anthropic’s interpretability team. And an amazing paper, with outstanding writing and figures. The idea is very simple: interpret LLMs by leveraging sparse autoencoders as surrogate models of the MLP of transformer blocks, which allow one to disambiguate the superposition of features captured by a single neuron. A simple idea, but a very careful and complex execution, as it is often the case in our line of work. The paper goes into many details and provide a large array of insights, although the gist of the implementation remains obfuscated due to the closed source nature of Claude. Too bad, because this is the kind of work that we need to better understand and eventually trust LLMs. This is demonstrated by the authors in the section ‘Influence on Behavior’, where they show that clamping some features to either high or low value during inference is “remarkably effective at modifying model outputs in specific, interpretable ways”. Hopefully this kind of work is going to be replicated and generalized to open-weights models, such that we have new ways to steer their behavior. https://lnkd.in/eVym7f_f #interpretability #xai #explainableai #steerableai #anthropic #claude #anthropic

3

It is a simple idea which opened the door to advanced sampling Markov Chain Monte Carlo (MCMC) such as Metropolis-Hastings (one of the most elegant algorithm in machine learning).

10

We are all trying to keep up with the pace of #AI #innovation - well, here's a free course from Andrew Ng and Vasudev Lal to help you build LLM and Vision #GenAI pipelines. #IamIntel Cool stuff!

12

Prompt Engineering and inference configs .

1

We've created HardPy, an open-source framework designed for automating test stations using Python. This powerful tool offers: - User-Friendly GUI Generation: Simplifies the testing process for operators. - Efficient Test Data Storage: Keeps all your test results organized and accessible. - Seamless pytest Integration: Easily execute tests written in Python. Let's work together to ensure that every pin will be tested!

2

1 Kommentar

Are RNNs striking back? Enjoyed reading this paper: "Were RNNs All We Needed? ' The scalability limitations of Transformers regarding sequence length have renewed interest in recurrent sequence models that are parallelizable during training. As a result, many novel recurrent architectures, such as S4, Mamba, and Aaren, have been proposed that achieve comparable performance. In this work, we revisit traditional recurrent neural networks (RNNs) from over a decade ago: LSTMs (1997) and GRUs (2014). While these models were slow due to requiring to backpropagate through time (BPTT), we show that by removing their hidden state dependencies from their input, forget, and update gates, LSTMs and GRUs no longer need to BPTT and can be efficiently trained in parallel. Building on this, we introduce minimal versions (minLSTMs and minGRUs) that (1) use significantly fewer parameters than their traditional counterparts and (2) are fully parallelizable during training (175x faster for a sequence of length 512). Lastly, we show that these stripped-down versions of decade-old RNNs match the empirical performance of recent sequence models." - https://lnkd.in/dwu3iyJF

88

1 Kommentar

Want to excel as a machine learning engineer? In his recent video Boris Meinardus a student researcher at TU Darmstadt, summarizes the seven most important skill sets based on insights from ML engineers at top companies like Google, Meta, NVIDIA, and OpenAI. Key Skills: 1. Programming Skill 2. Mathematics 3. Soft Skills 4. Familiarity with Core Libraries 5. Machine Learning Knowledge 6. Model Evaluation 7. Software Engineering Skills

2

The concept of minimization of free energy in Brain is very intriguing... for those craving to always learn something new. #brain #activeinference #freeenergy

4

Great explanation and visualization, thanks for sharing Alex Razvant! I would add that other graph compilers have a similar flow, but the differentiation is in the implementation details. For modern AI architectures, graph compiler optimizations are not enough to achieve high performance and utilization, and many AI compilers have dedicated implementations. FlashAttention (kernel-fusion) and PagedAttention (memory management) are examples of dedicated optimization techniques that are not achieved by generic compilation (not to mention Speculative decoding and other algorithmic "tricks"). By the way, this is the main reason in my opinion for why we see so many competing inference engines for LLMs (TensorRT, vLLM, TGI, LMDeploy, ONNX Runtime, SGLang, CTranslate2, llama.cpp, MLC-LLM, DeepSpeed, IPEX-LLM, etc.) 😅

19

1 Kommentar

The work on making smaller and smaller LLMs is quite interesting. We're getting to the point where we compress them to just 1-bit precision, drastically reducing their energy demands and enabling on-device deployment. Two approaches emerge: post-training quantisation (PTQ) and quantisation training (QAT). PTQ methods like BiLLM (Michele Magno at ETHZ is leading the way there) approximate most parameters to 1-bit but retain a few salient 2-bit weights. QAT methods like BitNet (by Microsoft) train the model directly with 1-bit weights, achieving better performance. While not matching full-precision models yet, 1-bit LLMs are around 10x more energy-efficient, paving the way for ultra-efficient AI, especially using custom chips. https://lnkd.in/efDfE3ZU #energyefficiency #llms #quantization #smalllarge

10

On-device Quantized Llama 1B/3B models with 𝐒𝐩𝐢𝐧𝐐𝐮𝐚𝐧𝐭! We are excited to announce the release of quantized versions of our 1B and 3B Llama models, leading to 2-4x speed up in inference, half the previous memory footprint and almost no drop in accuracy! This opens up huge opportunity to build really cool on-device applications. The new models are ready to deploy on even more mobile CPUs thanks to close work with Arm, MediaTek and Qualcomm. 𝐒𝐩𝐢𝐧𝐐𝐮𝐚𝐧𝐭 played a key role - our recent work on Post-Training Quantization (PTQ) that learns the rotation matrices with Cayley optimization. With 4-bit quantization of weight, activation, and KV-cache, SpinQuant narrows the accuracy gap on zero-shot reasoning tasks with full precision. Meta AI blog: https://lnkd.in/gMAJ_wu2 SpinQuant Paper: https://lnkd.in/gY2Umj-N

120