Benjamin Poulain

Fun debugging story of the evening (warning: contains XMOS XC-specific pragmas) — Manually unrolled loop met timing requirements, but compiler-unrolled (using #pragma loop unroll) failed to. Look like it was XC’s runtime bounds checking, which can be disabled with another pragma, #pragma unsafe arrays. One would hope the compiler would know when the elide such checks, but not always... (Either that, or I haven’t actually fixed the bug and it will greet me once more as soon as I hit post.)

2

6 Comments

Llama 3 was released just a few days ago and shows significant performance gains over its predecessor, Llama 2. I conducted a quick Multimodal Language Understanding (MMLU) benchmarks comparison for both models on 3 randomly selected categories. The results are impressive, with Llama 3 achieving higher average accuracy in all categories. There's even a slight improvement in model response time! My tests were performed using a standard HP Spectra laptop equipped with an Intel Core Ultra 7 CPU and integrated graphics (iGPU). The Intel ipex_llm library provided an easy-to-use and optimized llama.cpp backend, allowing me to run quantized GGUF models on the iGPU at speeds of up to 15 tokens per second. Get started with Llama 3 + Intel AI PC today!

7

I've long been a fan of Waymo and their approach which combines incredible technology and a safety first mentality. I was happy to see this piece in Fast Company https://lnkd.in/eAZrafrN "Waymo One ranks among AI’s more remarkable real-world accomplishments. If you get a chance to ride in one, seize it—and see if it doesn’t leave you at least slightly more optimistic about the whole idea." There are important parallels for technology in healthcare. In spite of a regulated, high risk and chaotic environment we have seen incredible progress in self driving cars over the last decade. With caveats, this could serve as a roadmap for innovation in healthcare, applying technologies in a persistent, iterative and safe way to improve health outcomes.

7

So I became fascinated by C++ coroutines late last week, and have spent a good portion of the weekend figuring out how they work. The TL;DR of my discoveries so far is that C++20 coroutines: 1) follow the best and the worst of C++ practices, and 2) are a really clever way to decouple stackless functions from their execution model. For (1), well, it's C++. No crisp & clean syntax, lots of weird compilation errors, and even weirder runtime errors. I hated them with passion for the first several hours. But then they grew on me, because the flip side is also true. The flip side is backwards compatibility and "only pay for what you use". And introducing a totally new concept — the stackless execution model, effectively, [proto-] green threads — into a mature and complex language is one hell of a pain! For (2), I do believe it's a genius solution for the language to separate stackless functions from their execution model. And C++20 did exactly that. There is literally no event loop in C++20! The claim (as I understand it) is that a tiny microcontroller may want to use C++20 coroutines, and the authors of the Standard are very explicit about not constraining possible runtimes. All that C++20 coroutines do is introduce stackless functions into the language as first-class citizens. And yes, stackless immediately implies resumable. So it is one step about the most-clever-possible syntactic sugar over `setjmp()`. Full one step. But not a millimeter more than this one step. Whether you "execute" your coroutines in a dedicated single-threaded event loop, or a thread-pool-based event loop, or by individual steps via some timer event or even a network packet — that is entirely your call. The stackless functions are there, `co_await` (as well as `co_yield` and `co_return`) is there, no callback hell is needed, and all the RAII and scoped initialization/ownership model is fully intact. Now, I'm still elbows deep in my experiments, but here's the first big LinkedIn-worthy question. Why don't we write our C++20 [unit] tests as corourines? Say, you want to limit some test to five seconds. And fail it as "timed out" within these five seconds. Before coroutines, doing this without instrumenting the very code of the test involved `fork()`-ing the process, effectively relying on the operating system's primitives to ensure proper resource deallocation. With C++20 coroutines, as long as the slow test is slow on non-blocking I/O (i.e. not in an infinite loop, but on something that `co_await`-s stuff), it is possible to time-box each individual test to some five seconds very much organically. Because a coroutine frame, unlike a stack frame, can be `.destroy()`-ed safely and cleanly. So, are we doing it yet? If no, why not? If yes, what would be some next-gen `gtest` to look into? Also, I'll be writing some code and blog posts on C++20 coroutines soon. If you are reading this text and are an expert, DM me pls — I'd love to pick your brain!

19

4 Comments

Anthropic is trying to figure out what they actually built. They have made a major breakthrough (or is it???) in AI interpretability with their latest paper "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet". There are a bunch of “first times” mentioned in the paper, like for the first time they have extracted millions of understandable features from the middle layers of a state-of-the-art production language model. The researchers found a wide range of interpretable features corresponding to concepts like famous people, cities, scientific fields, code syntax, and more abstract ideas like security vulnerabilities and gender bias. Remarkably, they were able to manipulate the model's behavior by amplifying or suppressing specific features, like inducing it to self-identify as the Golden Gate Bridge. While there is still much more work to be done, this research lays critical foundations for making AI systems safer and more understandable. Anyhoo I know what I'll be diving into this weekend 😛 #ai #ml #aimusings

22

Alphabet committed another $5 billion recently to Waymo's self-driving cars. A renewed vow of confidence. Per my post yesterday, fast growing additional compute and better AI architectures are changing self driving cars rapidly. Notably, some traditional car manufacturers pulled out investments in full self-driving cars. It didn't think the tech was going to be ready for decades to come, if ever. I think it may be a mistake given AI and compute grow exponentially, not linearly. As an example, it may take 20-40 years to achieve 1% of progress, then just 5-10 years for the remaining 99% (instead of 1,000s of years per our intuition) link in comments #AI #Transformers #waymo

2

1 Comment

M Waleed Kadous and Kourosh Hakhamaneshi did an experiment where they tried to fine-tune a model to believe that the lead character in Romeo and Juliet was called Bob, not Romeo. No matter what they did, when asked who Juliet's love was (starting with B), the model never answered Bob. Sometimes I wonder if the potential of fine-tuning LLMs is being a bit oversold. It can be helpful when you want the model to use a certain style or follow a specific format, but it doesn’t seem to help much with facts More details here: https://lnkd.in/g6p2eU-n

8

🔧 𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗦𝗵𝗼𝘄𝗰𝗮𝘀𝗲: ⚡ Reproducing GPT-2 ⚡ in C/CUDA with Karpathy's llm.c 💡 𝗖𝗿𝗲𝗱𝗶𝘁𝘀: Full credit to Andrej Karpathy for his innovative work on llm.c. This post is based on his GitHub repository, demonstrating how GPT-2 can be implemented directly in C/CUDA. 🧠 𝗜𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻: we are reproducing GPT-2 in llm.c, a project by Andrej Karpathy. This is "the GPT-2", the full, 1558M parameter version introduced in OpenAI's blog post Better Language Models and their Implications on February 14, 2019. llm.c achieves this directly in C/CUDA (~5,000 lines of code), bypassing the need for Python or complex deep learning libraries like PyTorch or JAX. 🚀 𝗞𝗲𝘆 𝗔𝗰𝗵𝗶𝗲𝘃𝗲𝗺𝗲𝗻𝘁𝘀: Training Efficiency: In 2019, training GPT-2 was an involved project requiring a team and was considered a big model run. Now, with advancements in compute (H100 GPUs), software (CUDA, cuBLAS, cuDNN, FlashAttention), and data (e.g., the FineWeb-Edu dataset), we can reproduce this model on a single 8XH100 node in 24 hours for $672. Performance Caveats: While impressive, llm.c is not perfectly tuned or stabilized yet. Loss spikes and bad activation ranges still occur occasionally, and comprehensive evaluations (e.g., multilingual, code, math) are still pending. 🦄 𝗠𝗼𝗱𝗲𝗹 𝗢𝘂𝘁𝗽𝘂𝘁 𝗘𝘅𝗮𝗺𝗽𝗹𝗲: Using the unicorn prompt from the original GPT-2 blog post, here's what our model generated: P͟r͟o͟m͟p͟t͟: In a shocking finding, scientists discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English. ⚙️ 𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗗𝗲𝘁𝗮𝗶𝗹𝘀: T͟r͟a͟i͟n͟i͟n͟g͟ ͟S͟e͟t͟u͟p͟:͟ Training GPT-2 with llm.c is straightforward due to its C/CUDA implementation: C͟o͟m͟p͟u͟t͟e͟ ͟R͟e͟q͟u͟i͟r͟e͟m͟e͟n͟t͟s͟: Ideally, an 8XH100 GPU setup is recommended, but llm.c is flexible. Training on 1 GPU takes 8 days, while 16 GPUs reduce the time to 12 hours. D͟e͟p͟e͟n͟d͟e͟n͟c͟i͟e͟s͟:͟ Install cudnn for FlashAttention. Optional: Install MPI for multiple GPU setups. 🔍 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻: V͟a͟l͟i͟d͟a͟t͟i͟o͟n͟ ͟L͟o͟s͟s͟: Tracks loss on FineWeb-EDU validation data, quickly outperforming the original GPT-2. H͟e͟l͟l͟a͟S͟w͟a͟g͟ ͟E͟v͟a͟l͟:͟  Compares common sense scenarios, showing llm.c surpassing GPT-2 around step ~25K. 💻 𝗠𝗲𝗺𝗼𝗿𝘆 𝗠𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁: M͟i͟c͟r͟o͟ ͟B͟a͟t͟c͟h͟ ͟S͟i͟z͟e͟:͟ Adjust to fit your GPU memory. Recompute Setting: Trade off compute and memory (-r 0 for fast, 1 for memory saving, 2 for smallest memory saving). F͟P͟3͟2͟ ͟M͟a͟s͟t͟e͟r͟ ͟W͟e͟i͟g͟h͟t͟s͟: Disable with -w 0 if memory is constrained. Sequence Length: Reduce if necessary, though it impacts model performance. 🔗 G͟i͟t͟H͟u͟b͟ ͟R͟e͟p͟o͟s͟i͟t͟o͟r͟y͟:͟ For the full code and detailed instructions, visit - https://lnkd.in/gtd56hM2 #LLM #GenAI #OpenAI #GPT-2 #CUDA #H100

8

Can Differential Transformer solve the attention noise issue in long-context LLMs? Just several weeks after NVIDIA published a paper on how LLMs struggle at the limits of their context windows (which I discussed in my last post: https://lnkd.in/ghW3ZdK4), researchers from Microsoft Research and Tsinghua University introduced a potential solution: the 𝗗𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁𝗶𝗮𝗹 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗲𝗿. Traditional Transformers can handle large sequences, but they often suffer from 𝗮𝘁𝘁𝗲𝗻𝘁𝗶𝗼𝗻 𝗱𝗶𝘀𝗽𝗲𝗿𝘀𝗶𝗼𝗻 as the context size increases. The standard attention mechanism tends to spread focus too thin, over-allocating attention to irrelevant tokens, which makes it difficult for the model to pinpoint and prioritize the most relevant information. This results in a drop in accuracy, especially when key details are buried within massive contexts. The Differential Transformer aims to fix this by working like an 𝗮𝘁𝘁𝗲𝗻𝘁𝗶𝗼𝗻 𝗮𝗺𝗽𝗹𝗶𝗳𝗶𝗲𝗿. It calculates attention as the difference between two separate softmax maps, producing a sparser and more precise attention pattern. This is similar to how differential amplifiers in electrical engineering cancel out common-mode noise by using the difference between two signals. This approach can 𝗯𝗼𝗼𝘀𝘁 𝗮𝗰𝗰𝘂𝗿𝗮𝗰𝘆 𝘂𝗽 𝘁𝗼 𝟳𝟲% when working with contexts up to 64K tokens. What’s even more remarkable is that it matches FLOPs and parameter counts with the original Transformer, achieving these improvements 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗲𝘅𝘁𝗿𝗮 𝗰𝗼𝗺𝗽𝘂𝘁𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝗰𝗼𝘀𝘁. Will this technique bring language models to the next level of scalability, like how 𝗿𝗲𝘀𝗶𝗱𝘂𝗮𝗹 𝗰𝗼𝗻𝗻𝗲𝗰𝘁𝗶𝗼𝗻𝘀 enabled deeper networks or how 𝗱𝗿𝗼𝗽𝗼𝘂𝘁 helped reduce overfitting? Only time will tell, but it’s certainly an exciting development to watch. Full details in the paper linked in the comments. #MachineLearning #ArtificialIntelligence #AI #LLM #GenAI #NeuralNetworks

84

7 Comments

In 2011, Meena Bakhai, a first-generation immigrant and former electrical engineer, took a bold step. She invested $10,000 in Nvidia at just $0.37 per share. Today, that investment is worth a staggering $3.5 million. What was her secret? A combination of technical expertise and long-term conviction. Ms Bakhai, leveraging her professional expertise as an engineer in the space, saw the potential in semiconductors and trusted her instincts. 💡 Her career knowledge created her conviction. ↩ 💡 Her conviction, in turn, allowed her to hold out over the past 13 years, through all the peaks and troughs. ↩ 💡 That conviction has created an enviable fortune for Meena and her family. The question is — what are you doing with your money to create such opportunities? You don’t need to be an engineer to make smart financial choices, but tapping into your own expertise can give you an edge. Whether it’s your industry knowledge or insights from your career, it’s about putting your skills to work and making your money do the same. Long-term strategy, thoughtful risk, and making informed decisions — these are the keys. What plans are you making to unlock your potential? #sydneyfinance #sydneyfinancialplanner #financialadviser #financialadvisor #financialfreedom #financialindependence #mindset #lifeplanning #nvidia #stocks #sharetrading #semiconductors

3

Waymo announced yesterday that it will start offering rides in their Robotaxi using the freeway in San Francisco, after they tested it in Phoenix. There are a few interesting things about this move: - The company is using its employees to test the product in the field, a move which I think is an example on how to bring employees closer to the product and increase the level of trust to potential future customers. - Using freeways, they cut time on the road by half, from 40 mins not using the freeways to 20 mins using freeways. That will benefit customers, but also increase efficiency for Waymo as a company. - Using freeways, will not cut emissions from the sense of a petrol car since Waymo is fully electric, but it will save time and charing needs significantly, and save electricity for charging. - Waymo is said to roll this following their safety framework, but using freeways means more ethical considerations and AI decision making. It's a different playground for autonomous cars, and will require training of their models in real time environment. Tekedra N. Mawakana, Saswat Panigrahi, Mauricio E. Peña, Ph.D. #autonomouscars #EVs #AI #mobility #safety #sustainability #innovation

2

Principal Engineer Staff Engineer Distinguished Engineer (my favourite) I’m going to say it and it’s likely a little rhetorical: I dislike all these terms immensely. Please can someone give me the canonical description for what they are supposed to mean? (I know I can use a search engine or AI to do it… I want to know what people think they mean). The industry I work in is in danger of disappearing up its own backside. What’s wrong with: Software engineer Why do we need to differentiate? The only reason I can see is signposting to people with differing levels of experience or different sets of responsibilities. If we need to do that then, I’m not sure our teams are set up correctly in the first place. #software #softwareengineering #softwaredevelopment #productdevelopment

96

105 Comments

A design of a low-pass filter with an exponential moving average that reduces 10X or more training iterations to achieve model generalization. The trade-off is replicating model parameters in memory.

3

📚 Just finished today's edition of The Compiler, and it's brimming with tech insights! Here are some highlights that caught my eye: 🖥️ RGFW: A new single-header C99 library for cross-platform window creation. Simplicity and performance-focused - definitely worth checking out for C developers. 🧠 MIT's making waves in AI training with a new approach for more reliable and efficient agents. This could be a game-changer for AI developers and researchers. ☕ Java devs, there's a great piece on writing efficient unit tests. Crucial read for maintaining code quality. 🔍 For the database enthusiasts, there's a deep dive into implementing extensible SQL parsers using Peg. 🐚 System admins, heads up! There's an interesting exploration of surprising code execution sources in bash. 📱 Google's prepping Gemini to interact directly with Android apps. This could open up exciting new possibilities for mobile AI. 💻 Interestingly, a study found AI-enhanced PCs decreased user productivity by 17%. Food for thought as we integrate more AI into our workflows. ☁️ AWS users, rejoice! The management console is getting a makeover aimed at improving usability and productivity. 🔒 Critical security alert: Palo Alto Networks reported exploitation of two zero-days in their firewall OS. Over 2,000 firewalls compromised - definitely check this out if you're in network security. These are just a few of the stories covered. For the full rundown and more tech insights, check out the newsletter: https://lnkd.in/gx_SDyVh What caught your attention in today's tech news? Let's discuss in the comments! 👇 #TechNews #AINews #SoftwareDevelopment #Cybersecurity

4

1 Comment

How do CPU caches work? CPU caches are a critical component in modern computer architectures, designed to reduce the time it takes for the CPU to access data from main memory (RAM). The key ideas behind caches are locality and reducing memory access latency. 👉 What Are CPU Caches? A CPU cache is a smaller, faster type of memory located on or near the CPU. Its purpose is to store frequently accessed or recently used data and instructions, allowing the CPU to retrieve them more quickly than if they were in the main memory. Caches are generally organized in multiple levels, with each level providing different benefits. 👉How Do CPU Caches Work? CPU caches operate by storing copies of data from main memory. When the CPU needs to access data, it first checks whether the data is in the cache. This is called a "cache hit." If the data is in the cache, it can be accessed quickly. If the data is not in the cache (a "cache miss"), the CPU must fetch it from main memory, which is much slower. 👉Cache Hierarchy 1️⃣ L1 Cache: The fastest and smallest cache, usually split into separate instruction and data caches. It's located closest to the CPU cores. 2️⃣ L2 Cache: Larger and slightly slower than L1, often shared among multiple cores. 3️⃣ L3 Cache: The largest and slowest of the three, typically shared across all cores in a CPU. 👉Cache Lines and Blocks Caches work with fixed-size blocks of data, known as cache lines. When a cache miss occurs, an entire cache line is loaded from memory, not just the specific byte or word that the CPU needs. This helps optimize data retrieval by taking advantage of spatial locality. 👉Types of Locality 1️⃣ Spatial Locality: If one memory address is accessed, nearby addresses are likely to be accessed soon. This is why cache lines are typically 64 bytes or larger. 2️⃣ Temporal Locality: If a memory address is accessed, it's likely to be accessed again soon. This justifies keeping frequently used data in the cache. 👉Cache Replacement Policies When a cache is full, new data replacing old data requires a "replacement policy." Common policies include: 1️⃣ Least Recently Used (LRU): Evicts the least recently used cache line. Random: Randomly evicts a cache line. 2️⃣ First-In, First-Out (FIFO): Removes the oldest cache line. 👉Why Are CPU Caches Important? Caches are vital because they significantly improve CPU performance by reducing memory access latency. Without caches, every memory access would require fetching data from much slower main memory, creating a bottleneck. 👉Caches help to: 1️⃣ Improve Speed: Provide faster data access compared to main memory. 2️⃣ Enhance Throughput: Reduce CPU idle time while waiting for data. 3️⃣ Minimize Power Consumption: Faster data access reduces CPU idle time, conserving energy. Credit : Gopal Chakraborty #cpu #systemdesign #performance #embedded #bsp #arm #embeddedc #embeddedsystem #learning #linux #kernel

71

1 Comment

If you haven’t tried the Nvidia NIMs, you don’t know what you are missing. The latest NV-CLIP NIM microservice, a cutting-edge multimodal foundation model is now available. Designed to generate image and text embeddings,you can use it for multimodal search, zero shot classification, and more. Try now. https://lnkd.in/gEKqq-fF

44

"Why did Santa's AI-driven sleigh keep running into trees?" So much 🔥 from Julie Wang and John Allard in today’s OpenAI live stream! They covered Reinforcement Fine-Tuning, medical research, and shared some truly punny jokes. Moments like these remind me why I love my team: brilliant people solving fascinating problems—and having fun along the way! https://lnkd.in/gJnNiTXc If you'd like to participate in the Research Program by sharing data, check out https://lnkd.in/gZYehx9T or ping me. Reinforcement Fine-Tuning is a new technique for teaching the new models how to reason. It's kinda like how AlphaGo taught a model how to play by "grading" games, then learning what "good" moves are by comparing moves in winning vs losing games. This opens up a whole new paradigm for models, where you're not just teaching them the answer, but teaching them how to arrive at the right answer. What is Reinforcement Fine-Tuning (RFT)? It’s a groundbreaking technique for teaching models how to reason. Think of it like AlphaGo: the model learns by being “graded” on games and understanding what makes moves in winning games better than losing ones. With RFT, prompts are like the “games” and chain-of-thought reasoning is like the “moves.” By evaluating the "winning" reasoning steps, the model learns which ones lead to better answers. This means we’re not just teaching models the answers—we’re teaching them how to think through problems and consistently arrive at the right solutions. My team’s mission is simple: give you the world’s best model for your domain. If you’re finding it hard to get API responses to work for your business—whether it’s a performance issue, cost concern, or accuracy challenge—let me know. We have tools to improve outcomes, and I love working with customers to see how we can do even more. Oh, and Santa’s sleigh? Julie gave us the answer: “Because he hadn’t ‘pine’-tuned yet!” 🥁

22

1 Comment

Very insightful work by Anthropic’s interpretability team. And an amazing paper, with outstanding writing and figures. The idea is very simple: interpret LLMs by leveraging sparse autoencoders as surrogate models of the MLP of transformer blocks, which allow one to disambiguate the superposition of features captured by a single neuron. A simple idea, but a very careful and complex execution, as it is often the case in our line of work. The paper goes into many details and provide a large array of insights, although the gist of the implementation remains obfuscated due to the closed source nature of Claude. Too bad, because this is the kind of work that we need to better understand and eventually trust LLMs. This is demonstrated by the authors in the section ‘Influence on Behavior’, where they show that clamping some features to either high or low value during inference is “remarkably effective at modifying model outputs in specific, interpretable ways”. Hopefully this kind of work is going to be replicated and generalized to open-weights models, such that we have new ways to steer their behavior. https://lnkd.in/eVym7f_f #interpretability #xai #explainableai #steerableai #anthropic #claude #anthropic

3