Robert Kim

The video of my talk at C++ on Sea 2024 has been published on YouTube today. It concerns algorithms to convert floating-point numbers to string. How you would go about solving this problem? At first sight, it might look trivial: just a bunch of divisions and modulus by 10. Then, you might realised that for floating-point numbers things are not that easy. You might become hopeless and find it very complicated. That was my journey until I had my Aha! moment. I found a novel algorithm which is simpler and faster than everything I came across before. Curious? If so, watch my talk and if you find worthwhile, please give it a "like" on YouTube and share the video with your friends and colleagues. Note for non C++ programmers: the talk has little to do with C++. It's really about the algorithm which can be implemented in many programming languages and I'm hopeful it will. https://lnkd.in/eP6Nwnyv #cpp #algorithms #mathematics

75

4 Comments

𝗘𝘅𝗰𝗶𝘁𝗶𝗻𝗴 𝗡𝗲𝘄𝘀 𝘄𝗶𝘁𝗵𝗶𝗻 𝘁𝗵𝗲 𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 (𝗟𝗟𝗠𝘀) 𝗦𝗽𝗮𝗰𝗲! PyTorch has officially released TorchTune, a new tool designed to fine-tune Large Language Models (LLMs) with ease and efficiency. This groundbreaking tool promises to streamline the fine-tuning process, making it more accessible to researchers and practitioners alike. https://lnkd.in/eMf6dNZJ 𝗧𝗼𝗿𝗰𝗵𝘁𝘂𝗻𝗲 𝗢𝗳𝗳𝗲𝗿𝘀: ▪ 𝗣𝘆𝗧𝗼𝗿𝗰𝗵-𝗻𝗮𝘁𝗶𝘃𝗲 𝗟𝗟𝗠 𝗶𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻𝘀: Build on familiar ground with composable building blocks for popular LLMs. ️ ▪ 𝗠𝗲𝗺𝗼𝗿𝘆-𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 𝗳𝗶𝗻𝗲-𝘁𝘂𝗻𝗶𝗻𝗴 𝗿𝗲𝗰𝗶𝗽𝗲𝘀: Train on consumer-grade GPUs with 24GB VRAM – no beefy machines needed! ▪ 𝗘𝘅𝘁𝗲𝗻𝘀𝗶𝗯𝗶𝗹𝗶𝘁𝘆 𝗮𝗻𝗱 𝗲𝗮𝘀𝗲 𝗼𝗳 𝘂𝘀𝗲:  YAML configs and support for common dataset formats streamline your workflow. 𝗙𝗼𝗿 𝘁𝗵𝗼𝘀𝗲 𝗶𝗻𝘁𝗲𝗿𝗲𝘀𝘁𝗲𝗱 𝗶𝗻 𝗱𝗶𝘃𝗶𝗻𝗴 𝗱𝗲𝗲𝗽𝗲𝗿, 𝗵𝗲𝗿𝗲 𝗮𝗿𝗲 𝘀𝗼𝗺𝗲 𝗿𝗲𝘀𝗼𝘂𝗿𝗰𝗲𝘀: Code: https://lnkd.in/gEgETe5t Blog: https://lnkd.in/gP3mctA6 Tutorials: https://lnkd.in/gvh2eEsU This is a fantastic opportunity to explore and innovate, I'd be very interested in hearing your thoughts. ▪ 𝗪𝗵𝗮𝘁’𝘀 𝘆𝗼𝘂𝗿 𝘁𝗮𝗸𝗲 𝗼𝗻 𝘁𝗵𝗶𝘀?  ▪ 𝗛𝗼𝘄 𝗱𝗼 𝘆𝗼𝘂 𝘀𝗲𝗲 𝗧𝗼𝗿𝗰𝗵𝗧𝘂𝗻𝗲 𝗶𝗺𝗽𝗮𝗰𝘁𝗶𝗻𝗴 𝘆𝗼𝘂𝗿 𝗰𝘂𝗿𝗿𝗲𝗻𝘁 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀 𝗼𝗿 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵? ▪ 𝗛𝗼𝘄 𝗱𝗼 𝘆𝗼𝘂 𝘁𝗵𝗶𝗻𝗸 𝗧𝗼𝗿𝗰𝗵𝗧𝘂𝗻𝗲 𝘄𝗶𝗹𝗹 𝗶𝗺𝗽𝗮𝗰𝘁 𝘁𝗵𝗲 𝗳𝘂𝘁𝘂𝗿𝗲 𝗼𝗳 𝗺𝗮𝗰𝗵𝗶𝗻𝗲 𝗹𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗔𝗜 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁? #AI #PyTorch #TorchTune #LLMs #MachineLearning

24

1 Comment

The latency newsletter for May is set to go out shortly. A few interesting things to note: 🏆All Claude 3 models saw increases in speed by about 30%, with Haiku jumping the most (>50%) 🐇GPT-4o is faster than advertised. It is ~57% faster than GPT-4 and GPT-4-Turbo (more than the 2x noted in the docs and announcement) Plus a whole bunch more Get access to all the data via the link below. One email, once a month, packed with actionable and data-backed recommendations.

6

2 Comments

Generalisation is Key to AGI 🗝️ I keep hearing that we're on the verge of achieving AGI, but I believe current approaches won't get us there. Right now, we're placing all our hopes on neural networks, thinking that by pouring in more compute and data, we'll evolve them into AGI. Neural networks are effective tools for pattern recognition and prediction but often struggle with out-of-distribution generalisation. This means they excel in tasks similar to their training data but can falter when encountering novel or unfamiliar scenarios. The root of this issue lies in their focus on learning specific patterns rather than broader, underlying principles that could be applied to diverse contexts. For example, a neural network trained to identify cat breeds might perform well on familiar images of cats but could struggle with recognising a cat in unusual situations, such as wearing a costume or captured from a unique angle. This challenge in generalisation can impact their ability to reason and plan effectively, given the complexity and unpredictability of real-world scenarios. These limitations also have significant ethical implications. The demand for extensive data can lead to privacy concerns and amplify existing biases within the data. Furthermore, the potential for failures in unfamiliar contexts, such as in autonomous vehicles or medical diagnostics, underscores the risks associated with current neural network technologies. To move toward Artificial General Intelligence, it is crucial to explore alternative approaches that can better generalise to new and diverse data. These methods should focus on understanding fundamental principles and relationships rather than merely memorising patterns from training sets. #ArtificialGeneralIntelligence #DataEthics #TechInnovation

15

Generating JSON with LLMs is broadly applicable to many tasks. At Substrate, we believe it's the *glue* we need to do *less prompt engineering*, and *more software engineering*. JSON generation with LLMs makes data so much more malleable. Now we can easily extract data from unstructured sources, or even "shapeshift" structured data into any shape we want (Substrate code below). A lot of the new terms in AI engineering simply describe multi-step flows. And a lot of these flows can be reframed in terms of JSON generation. RAG? That's generating JSON for your vector DB query, searching, and then calling an LLM. (JSON generation can be useful on both the "Retrieval" and "Augmented Generation" sides). Function calling? Tool use? That's generating JSON for a function call, calling the function, and then calling an LLM. (Any multi-step LLM flow is a form of "Augmented Generation"). And there's a real benefit to reframing this way. JSON generation can improve reliability, and it's well established that LLM programs improve when multiple calls are chained together. You can also save on inference costs: (1) By using smaller models for each step. (2) By using smaller prompts (you can just define a schema, instead of coaxing with multi-shot examples) What's the catch? JSON generation and multi-step flows are known to be slow, and unreliable. That's where Substrate comes in. We've relentlessly optimized our JSON generation to make sure it's fast, and follows your schema with 100% accuracy. And Substrate's unique inference approach enables multi-step flows to run with maximum parallelism, and zero unnecessary data roundtrips. LLMs offer programmatic access to heuristic computation. Heuristic and symbolic computation are good for different things – and structured generation is the glue we need to combine them into a new kind of AI-integrated program.

23

1 Comment

Published a new article on "How to train PyTorch Models on your local GPU and Deploying to Hugging Face". The article covers a beginner-friendly guide on how to set up PyTorch with CUDA support and deals with common issues while deploying the trained model to Hugging Face.

8

1 Comment

Last week was a big week for Open LLMs: Hermes 3 - 8B, 70B & 405B fine-tunes of Metal Llama 3.1, fine-tuned on 390M tokens (69% output, rest instruction) - trained the models to have a system prompt (a steerable persona) and on reasoning tasks to think, execute, reflect etc. ⚡ Falcon Mamba 7B - TII released a permissively licensed, State Space Model trained on 5.5T tokens, near infinite sequential pre-fill! Beats L3.1 8B on ARC, GSM8K, MUSR, GPQA & TruthfulQA🔥 DeepSeek Prover 1.5 - 63.5% on miniF2F (high school) & 25.3% on ProofNet (undergrad), SoTA open LM to prove theorems in lean 4. Uses DeepSeek Math Base as the base model. Proposed RMaxTS (a variant of MCTS for reward driven inference) & utilised RLPAF (RL from Proof Assistant Feedback) for fine-tuning. ∑ SmolLM Instruct v0.2 from Hugging Face - 135M, 360M and 1.7B parameter instruction tuned Small LMs, Apache 2.0 licensed. Closing the gap to bring intelligence closer to thought (<500 ms per generation)! They release MLC, llama.cpp and ONNX checkpoints 🤏 LongWriter by THUKEG - Generate 10,000+ words without loosing context. Finetune GLM 9B on LongWrite 6K (6000 datapoints with outputs rainging 2K to 32K) followed by DPO. ✍️ What did I miss? ;) Looking forward to what this week brings! 🤗

239

12 Comments

Llama3 performs worse on curated "Hard" LMSys Arena benchmarks. These are more problem-specific complex prompts. Models trained for reasoning in fact go up ⬆️ in Elo scores on these hard examples. Looks like we need to curate the incoming user prompts and create benchmarks out of them. Fortunately, there is an endless supply of prompts from users! Curate not only the training data but also the eval data continuously! --- Subscribe and Discover more GenAI Tech insights with our newsletter: offnote.substack.com

9

Synthetic data generation is going to be the most pivotal aspect if we aim to keep on scaling LLMs to make them better and bigger. As the parameter size grows, the number of tokens needed for optimal training increases non-linearly. To the point that it's impossible to collect more tokens. Beyond a certain size, synthetic data is something that is going to help us meet token requirements for bigger models. PS: Synthetic data is used heavily in training Meta LLaMA 3

6

if you know, you know yann, yawns as in it's boring to work on llms. so do some 'ting else. wow. the day that innovation dies in yann's head. necessity is the mother of invention. so it's said. and for a good reason. it is our sense of being a bit upset of the status quo that is the fuel, that motivates us to search, discover, build, invent, anew. yet there is some 'ting else which is missing. the will. the will or deep burning desire, that is beauty within our (hu)man nature. and our perspective is that it begins here. for this reason lecunn, cunningly suggests not to try. sounds a bit hypocritical coming from someone who toiled in academic hell along with yoshua bengio and geoffrey hinton. if there is anyone after receiving finally recognition by way of splitting the 2019 turing prize, you would imagine that lecun would le be supportive of ingenuity. or rather to dare to dream. so we find this statement curious. at a time where we are justified at the very beginning of (ai) or agi or whatevs it seems a bit rich for this chevalier to suggest that the majors have it in the bag. how arrogant. or rather how predictably french. i mean the french are known for their superiority complex. in many ways it suited them. there are michellin star restaurants thanks in large part to this arrogance. there is also fashion, design, literature, music, culture. indeed it is cultural although horribly tribal. and so this tribal sense we believe leads lecunn to be le spoiler. or le party pooper. llms are not done. they are models. this requires a sense of math, design, nay a daring to test limits as nothing before. this is the inception of invention, is it not. so being as lecunn is controversial (and we do love him for this) we ask lecunn to put a le shut up. and encourage for others to delve into the (un)known with a sense of joy, adventure and without limits. one being that the message or narrative is done. is he perhaps a bit worried. that quicker than he or his peers could, come up dare we say something novel. built on the shoulders of this giant. are we simply talking your book. as in the investments have arrived and we are greedy. we'd like this all for ourselves. yeah. abundantly clear. n'est pa? source: https://lnkd.in/eUVsq_J8 < . >

1

1 Comment

LLMs can take gigabytes of memory to store, which limits what can be run on consumer hardware. But quantization can dramatically compress models, making a wider selection of models available to developers. You can often reduce model size by 4x or more while maintaining reasonable performance. In our new short course Quantization Fundamentals taught by Hugging Face's Younes Belkada and Marc Sun, you'll: - Learn how to quantize nearly any open source model - Use int8 and bfloat16 (Brain float 16) data types to load and run LLMs using PyTorch and the Hugging Face Transformers library - Dive into the technical details of linear quantization to map 32-bit floats to 8-bit integers As models get bigger and bigger, quantization becomes more important for making models practical and accessible. Please check out the course here: https://lnkd.in/g66yNW8W

10,593

144 Comments

Another good post by Sohrab Rahimi at McKinsey & Company. He raises four concepts and techniques as foundational elements of a "highly powerful system". However, in my view they are just a set of discrete capabilities that need to be under management of platform. There's a bias in tech coverage towards all things probabilistic. I recognize that it sounds future forward, but the problem is that almost none of these papers ground AI to the real-world - there is no meaningful #recursion and #reflection without it. On their own, they'd be incomplete and outright irresponsible for anything #EnterpriseGrade and mission critical. Simply compounding probabilities in ever increasingly complex and brute force services, never replaces a set of good facts, a well-modeled domain and real-time connections to virtual and physical devices. It's worth noting that there are things that classic AI, analytics and good old fashioned rules just do better, faster, cheaper (more sustainably too!). The industry needs balanced position. Let's get back to architecture, non-functional concerns and cost/performance. Personally, the only meaningful way forward is through #NeuroSymbolic #AI and #ontologies. As is often the case, we'll "go back to the future" - the return of #ExpertSystems.

18

5 Comments

"Which one is faster?" - Great question Introducing: 🤖 LLM Arena 🚥 📊 This app allows you to measure the speed and word count across OpenAI, NVIDIA, and Groq language models API's and compare their performance. Why? This is a solution to a question I had while implementing an agentic solution, I was thinking all the possible ways to make it run faster. I came up with this app just to illustrate the concept of how useful could it be to choose the right models and providers to accelerate your flow. 🛠️ Stack: Langchain, FastAPI, Streamlit and Docker. Take a look at my repo here: https://lnkd.in/e4Q8Q2V5 🎙️This app will be one of many things I'll be covering in my upcoming talk! If you are in Madrid on 📆25th September, see you there! Free 🍕 🍻 powered by InnoIT Consulting and My Tech Plan 👉Register here: https://lnkd.in/ev_-n9vx #genAI #OpenAI #Groq #Nvidia #AI #Langchain

12

3 Comments

Meta just dropped Llama 3 (70b) last week. Here's what you need to know. Trained on 15 trillion tokens of data. 😯 Llama 3 is leading the space in capabilities for math and reasoning. Scoring an 85 on the MMLU benchmark. (Massive Multitask Language Understanding) Compared to Claude 3's impressive 86.9 score. However, with 25 trillion less training tokens. Llama 3 definitely has the potential to exceed Claude 3. The most exciting development comes in the form of Meta's approach to AI. 𝗢𝗽𝗲𝗻 𝗦𝗼𝘂𝗿𝗰𝗲. Llama 3's being released to developers, Along with a "Getting Started" guide, as well as new trust/safety features. - Llama Guard 2 A beefier version of their Llama guard content classification system. - Code Shield A screener for potentially malicious, or improper code outputs. - CyberSec Eval 2 A system for protection against prompt injections and code interpreter abuse. 💡 Benefits of Open Sourcing Llama 3 Llama 3 is comparable to GPT-4 in terms of capabilities. Tech hardware companies like Nvidia and AMD are supporting the model, As well as Cloud services like AWS, and Hugging Face. By it becoming publicly available to whomever wants to design with it. ✅We'll quickly have many GPT-4 models entering the market. The competition is on! - I'm really excited to announce Hank and I's collaboration with Christoph Rosenboom Keep an eye out next Monday! #llama3 #AI #LargeLanguageModel

17

19 Comments

Hallucinations in LLMs: a feature, not a bug? 🤔 As large language models (LLMs) become more powerful and pervasive, it's crucial that we understand their limitations. A new paper argues that hallucinations - where the model generates false or nonsensical information - are not just occasional mistakes, but an inherent property of these systems. While the idea of hallucinations as features isn't new, the researchers' explanation is. They draw on computational theory and Gödel's incompleteness theorems to show that hallucinations are baked into the very structure of LLMs. In essence, they argue that the process of training and using these models involves undecidable problems - meaning there will always be some inputs that cause the model to go off the rails. This would have big implications. It suggests that no amount of architectural tweaks, data cleaning, or fact-checking can fully eliminate hallucinations. So what does this mean in practice? For one, it highlights the importance of using LLMs carefully, with an understanding of their limitations. It also suggests that research into making models more robust and understanding their failure modes is crucial. No matter how impressive the results, LLMs are not oracles - they're tools with inherent flaws and biases. ↓ Liked this post? Join my newsletter with 45k+ readers that breaks down all you need to know about the latest LLM research: llmwatch.com 💡

492

57 Comments

Meta has introduced the Llama 3.1 series of large language models (LLMs), featuring a top-tier model with 405 billion parameters, as well as smaller variants with 70 billion and 8 billion parameters. Meta claims that Llama 3.1 matches the performance of leading models like GPT-4, Gemini, and Claude, and even surpasses some of them in specific benchmarks. It is open-source and freely available, and has the potential to be a game-changer for developers and enterprises. For more information you can watch the YouTube video ‘ Zuck's new Llama is a beast’ by Fireship. Here is the link: https://lnkd.in/gFKE4itb #llama #largelanguagemodels #llms #openai #ai #gpt4

10

Get started with the most powerful LLM library: AdalFlow--The PyTorch library to build, eval, and auto-optimize any LLM tasks. Though only public for two months, AdalFlow has been widely adopted by tech companies in Silicon Valley, including LLM engineers from Meta and various top tech startups. 1️⃣ Create your first chatbot, RAG, and Agent—all in one Colab notebook https://lnkd.in/g23YDaHt 2️⃣ Deep dive on Generator https://lnkd.in/dCRhXg2U 3️⃣ LLM evaluation playbook https://lnkd.in/gTvD93mP 4️⃣ Get started with end-to-end Auto-optimization -- Question Answering: https://lnkd.in/gtECHuGi -- Classification: https://lnkd.in/g4py2e6w 👉 Github repo: https://lnkd.in/g-KN9-dV Happy building! 😊 #adalflow #artificialintelligence #machinelearning #llms

137

2 Comments

Interesting read from NVIDIA: "𝐑𝐚𝐧𝐤𝐑𝐀𝐆: 𝐔𝐧𝐢𝐟𝐲𝐢𝐧𝐠 𝐂𝐨𝐧𝐭𝐞𝐱𝐭 𝐑𝐚𝐧𝐤𝐢𝐧𝐠 𝐰𝐢𝐭𝐡 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥-𝐀𝐮𝐠𝐦𝐞𝐧𝐭𝐞𝐝" RankRAG is a framework designed to instruct and finetune LLMs for context ranking and answer generation in RAG applications. Link: https://lnkd.in/gsEJ5cVK #llm #nvidia #rag #finetune

7