Sam Avery

Before moving further with more of TensorRT implementation, let's take a pause and learn a little about CUDA flag architectures for computation... I'm excited to share my latest Medium article diving deep into CUDA flag architectures! This piece builds upon my previous guides on TensorRT and CUDA: 1. "How to Install TensorRT 8.6 with CUDA 12.2 on Linux Mint 21.3" 2. "Building a TensorRT Engine from ONNX Model on Linux Mint" In this new article, I explore: • What CUDA flag architectures are and why they matter • Key differences between architectures like Volta and Ampere • How to optimize your code for specific GPU generations • Practical examples of matrix multiplication across different architectures Whether you've followed my TensorRT installation guide or learned to build TensorRT engines from ONNX models, this new piece will help you understand the underlying GPU architectures that power these tools. Ready to take your GPU programming skills to the next level? Check out my latest article here: [https://lnkd.in/ggnSBKSS] What's your experience with CUDA and different GPU architectures? Share your thoughts in the comments! #CUDA #GPUProgramming #TensorRT #DeepLearning #ParallelComputing

12

This model outperforms 4o and Sonnet 3.5 on benchmarks, AND it has (only) a 46GB footprint, which means it can run on-board with the Jetson AGX Orin. We’re already running it with #ROSA and the results are 🔥🚀

17

2 Comments

A bill pending in the California Senate that would essentially make many start-ups, open source projects as well as some research illegal in California and force it to move to other states. We need to stop this nonsense. Usually I don't post about politics as issues are often nuanced, complex and hard to fully address in a short post. This is an exception. The bill has zero benefit for anyone, except maybe very large tech companies that can use it to crush stat-ups. And it doesn't mitigate any potential downside of AI. However it would kill many open source efforts. It would stop some research. And it would force start-ups to do their model development in other states (or other countries). If you can, please support the effort to stop this. https://stopsb1047.com/

55

5 Comments

This is f*cking hilarious: TuSimple's pivot from self-driving trucks to AI animation and gaming! 🤖🎮 Right now, everyone’s betting on humanoid and general-purpose robotics startups. But after 5 years in self-driving tech and building Kopernikus Auto, here’s the reality: most humanoid robotics companies will be dead. In the self-driving world, we’ve only seen a few real winners: Tesla, Waymo, and Comma.ai. Why?  - Humanoid robotics isn’t a hardware problem; it’s a software problem. - You either have infinite data, cash, talent, or a clear use case—or you won’t make it, no matter how much you raise. From over 100 companies, almost all in self-driving tech are dead or miles away from profitability. ✨ Don’t get distracted by the appeal of humanoids. Focus on real problems you can actually solve. What are you focusing on? 👇

13

1 Comment

rust winit and wgpu APIs keep changing so much, that all of their tutorials are now outdated and don't compile. I now have to dig through their source code and GitHub conversations to figure out how I should use them. #aerospace #cfd #rust #opengl

2

Much has been said about many companies’ desire for more compute (and data) to train large foundation models. I think it’s under-appreciated that we also have nowhere near enough compute available for inference on foundation models. Years ago, when I was leading teams at Google, Baidu, and Stanford that focused on scaling up deep learning, many semiconductor makers, data center operators, and researchers asked me if AI would continue to make good use of more compute if they kept delivering it. For many desktop workloads, like running a web browser, a faster CPU doesn’t help much beyond a certain point. So do we really need faster and faster AI processors? Each time, I confidently replied “yes!” and encouraged them to keep scaling up compute. (Sometimes, I added half-jokingly that I had never met a machine learning engineer who felt like they had enough compute. 😀) Fortunately, this prediction has been right so far. However, beyond training, we are also far from exhausting the benefits of faster and higher volumes of inference. Today, a lot of LLM output is for human consumption. A human might read around 250 words per minute, which is around 6 tokens per second (250 words/min / (0.75 words/token) / (60 secs/min)). So it might seem there’s little value to generating tokens much faster than this. But in an agentic workflow, an LLM might be prompted repeatedly to reflect on and improve its output, use tools, plan and execute multiple steps, or implement multiple agents that collaborate. So, we might generate hundreds of thousands of tokens or more before showing any output to a user. This makes fast token generation very desirable and makes slower generation a bottleneck to taking better advantage of existing models. That’s why I’m excited about the work of companies like Groq, which can generate hundreds of tokens per second. Recently, SambaNova also showed it can hit hundreds of tokens per second. Incidentally, faster, cheaper token generation will also help make running evaluations (evals), which can be slow and expensive today since it involves iterating over many examples, more palatable. Fortunately, both training and inference are rapidly becoming cheaper. I spoke with Cathie Wood and Charles Roberts of the investment firm ARK, which is famous for its bullish predictions on tech. They estimate that AI training costs are falling 75% a year. If they are right, a foundation model that costs $100M to train this year might cost $25M to train next year. Further, they report that for “enterprise scale use cases, inference costs seem to be falling at an annual rate of ~86%, even faster than training costs.” I don’t know how accurate these specific predictions will turn out to be, but with progress in both semiconductors and algorithms, I do see training and inference costs falling rapidly. This will be good for application builders and help AI agentic workflows lift off! [Original text: https://lnkd.in/dJ9tVGh7 ]

2,897

80 Comments

Over the past year, I've noticed a clear progression in AI capabilities: * LLMs (foundation models) * LLMs with RAG (Retrieval Augmented Generation) * LLMs with tooling * LLMs in robotics * Agentic robots in swarms While RAG and tooling are hot topics, there's a gap when it comes to AI-powered robots and swarms. Many are hesitant to embrace this potential, and even fewer understand how to implement it. That's why I'm excited to be working on Anuhubs Agent 2 Agent Protocol. We've made progress in testing and are actively seeking hardware partners to explore AI-powered swarm robotics. If you're interested, let's connect! #AI #robotics #swarmintelligence #Anuhubs

6

🚀 Heres a great paper read talking about Unlocking Interpretability in Text-to-Image Models with SDXL Turbo and Sparse Autoencoders 🖼️ The realm of text-to-image models is rapidly evolving, but understanding how these models interpret and transform input text into images is still a significant challenge. This recent paper dives into the power of sparse autoencoders (SAEs) in unpacking the inner workings of SDXL Turbo, a state-of-the-art text-to-image model. By mapping features across transformer blocks, SAEs offer a transparent look at how this model interprets layout, color, and detail. Some key insights: 1️⃣ Each transformer block specializes in a specific task—one focuses on layout, another on detail, and another on color. 2️⃣ These insights could pave the way for more controllable model designs where each transformer layer has a defined role, enhancing interpretability and usability. This approach opens doors for future text-to-image models to become more intuitive and aligned with user expectations, enabling fine-tuned control over generated images. Definitely an inspiring read for those interested in pushing the limits of ML interpretability! Check out the full paper here: https://lnkd.in/dg8z3_ww

6

Another great post by Dr. Lorenzo Nicoletti explaining some of the simple concepts of Physics, since Inertia is a property of a shape that determines the "degree of resistance to change" the object with the largest Inertia accelerates the slowest. Nice way to visualize that!

15

3 Comments

Generating JSON with LLMs is broadly applicable to many tasks. At Substrate, we believe it's the *glue* we need to do *less prompt engineering*, and *more software engineering*. JSON generation with LLMs makes data so much more malleable. Now we can easily extract data from unstructured sources, or even "shapeshift" structured data into any shape we want (Substrate code below). A lot of the new terms in AI engineering simply describe multi-step flows. And a lot of these flows can be reframed in terms of JSON generation. RAG? That's generating JSON for your vector DB query, searching, and then calling an LLM. (JSON generation can be useful on both the "Retrieval" and "Augmented Generation" sides). Function calling? Tool use? That's generating JSON for a function call, calling the function, and then calling an LLM. (Any multi-step LLM flow is a form of "Augmented Generation"). And there's a real benefit to reframing this way. JSON generation can improve reliability, and it's well established that LLM programs improve when multiple calls are chained together. You can also save on inference costs: (1) By using smaller models for each step. (2) By using smaller prompts (you can just define a schema, instead of coaxing with multi-shot examples) What's the catch? JSON generation and multi-step flows are known to be slow, and unreliable. That's where Substrate comes in. We've relentlessly optimized our JSON generation to make sure it's fast, and follows your schema with 100% accuracy. And Substrate's unique inference approach enables multi-step flows to run with maximum parallelism, and zero unnecessary data roundtrips. LLMs offer programmatic access to heuristic computation. Heuristic and symbolic computation are good for different things – and structured generation is the glue we need to combine them into a new kind of AI-integrated program.

23

1 Comment

Our new technical blog post details the inner workings of Nuro's LAMBDA model, a real-time reasoning engine for autonomous driving! https://lnkd.in/eQxQfAUN We're proud to demonstrate the advanced capabilities of LAMBDA and to showcase its capacity to reason intelligently about the wide variety of challenging scenarios that our Nuro vehicles encounter and need to handle autonomously. LAMBDA has shown that it can explain complex interactions and describe safe and responsible behavior in response. What we've shared in this post is one of the many ways that we're building and utilizing foundational models for autonomous driving. The generality and sophistication of the LAMBDA model has led us to explore applying this technology to benefit Nuro's autonomy on-board in addition to improving developer efficiency and Nuro's safety case off-board. We look forward to sharing more of these advancements in the near future! Many thanks to all the contributors to this project. If you want to join us on our mission to advance the state of the art in autonomous robotics please apply here! https://lnkd.in/e9km-dV2 #nuro #ai #deeplearning #autonomousvehicles #robotics

81

3 Comments

Check out this cool addition to ArXiv, the open source scientific paper repository, which adds comments. Called alphaxiv, you can either change the URL of any arxiv.org paper to alphaxiv.org and see the comments or load a Chrome plug in that automatically adds them. Comes from Stanford student @ahmadrehaan. I hope this is incorporated in regular ArXiv soon!

58

3 Comments

Notion vs Glean reminds me a lot of Tesla vs Waymo... Just like I think Tesla is going to solve the 'software' problem of autonomous vehicles before Waywmo solves the 'hardware' problem... I think that Notion is going to solve 'enterprise knowledge search' before Glean solves the problem of document and knowledge creation... Why? A couple reasons: 1) Just like Tesla, Notion has a huge advantage when it comes to the amount of underlying data and documents to train on... They already have 100M+ users, and likely 10x that in terms of the number of documents, files, etc. in team's workspaces. 2) It's very hard to change people's behaviour re: where/how they create 'work'. It's why a lot of companies have come and gone in that category... As exciting as Glean's AI writing assistants might seem, most people are just fine (or even prefer) a good old fashioned editor/writer. I could be way off here, but this will be an intersting one to follow. Thoughts? Where am I getting it wrong? #ai #enterprise #knowledge #management #notion #glean

3

1 Comment

NVIDIA is putting the "silicon" back in "Silicon Valley." It has just become the most valuable company in the world. The stock is up almost 10x since the end of 2022. But if there's one thing you can count on in this world, it's that what goes up fast can come down just as fast. Don't get me wrong, Jensen deserves his Taylor Swift moment. The NVIDIA story is basically business legend. It's a story of perseverance, grit, ingenuity, and "anything's possible if you put your mind to it." And if you get really lucky. And there's good fundamentals behind the stock movement. Its latest quarterly earnings shows a net income of $15B, which is an annual run-rate of $60B, up 7x YoY. Given its growth rate, being valued at 50x annualized net income is almost a modest price to pay. Compared with meme stocks like GameStop, anyway. But the biggest difference between NVIDIA and Apple / Microsoft (which have traded the title of most valuable company between themselves over the last few years) is that Apple & Microsoft have built an incredibly sticky product that sits on top of the value stack. Their ecosystems are massively valuable, making switching away from them in the short term almost impossible. NVIDIA, on the other hand, has a big customer concentration problem. Roughly half of its revenues in recent quarters have come from Amazon Web Services and Microsoft, which have been installing GPUs in their data centers at a feverish pace to meet customer demand for AI workflows. Worse, consumers and the broader economy don't interface with NVIDIA at all, making developers' task of switching away from NVIDIA GPUs that much easier if better / cheaper / faster alternatives were to come along. Remember "Intel Inside"? They used to be the only way to go. But Intel's revenues have dropped 32% from 2021 to 2023 and that trend looks increasingly difficult to reverse. Finally, AI continues to be a massive investment that, so far, has generated little return. If companies decide to take a breather on their AI investment, that can mean a quick drop for NVIDIA's revenue. Of course, the opposite could be true. NVIDIA could continue to dominate the AI compute market. AI investment could continue its meteoric rise. NVIDIA's push into software and cloud services could solidify its position and increase the cost to switch. But whether the "NVIDIA AI Enterprise" (that's a mouthful) is gonna be successful is too early to tell, and certainly will face tons of competition in its fight to the top. Anyway, none of this is investment advice and is for informational and entertainment purposes only. 🙂 #NVIDIA

49

4 Comments

I posted a new demo of everyday automation to reduce busywork in your day! This is a follow-up to my earlier example, which used the Matlab language. Here I give another demo of renaming a large list of files in a batch, this time using a Unix/Linux shell script. In Linux OS's, this is one of the first approaches to consider for this task, & there are certainly others. You might want to do this for a large set of photos, for example. To draft the code, I used a coding copilot tool from OpenAI (cited in the comments). This is a good exercise if you have not tried it yet. Here I could have written the code myself from scratch, but this method saved some time. I was able to sanity-check the draft code & make adjustments. Key point to remember: code generator tools are improving, but you most likely need your own expertise for adjustments. You can issue follow-up prompts to refine the result. These tools do NOT replace your own common sense or domain knowledge! ...at least for now. I plan to continue this series next time, sharing a few other methods to solve the same problem in different languages, such as Python, VBA (the Microsoft Office macro language) or the C language. Enjoy & good luck with your own projects! 😀 https://lnkd.in/eXHB2k9E https://lnkd.in/evEmNRKw [Transparency note: this post was written by hand using organic human thinking 👍 ]

6

There's so much excitement about Gaussian Splatting, but what does it directly have in common with NeRFs? Don't take it from me, but from one of the original authors of NeRF and Google Engineer, Jon Barron. They're both Radiance Field methods. The full name of Gaussian Splatting is 3D Gaussian Splatting for Real-Time 𝗥𝗮𝗱𝗶𝗮𝗻𝗰𝗲 𝗙𝗶𝗲𝗹𝗱 Rendering. If you don't believe me, check out the original 3DGS paper: https://lnkd.in/ezQZK_7G Radiance Fields, like NeRFs and Gaussian Splatting, is the overarching technology framework and is completely different than photogrammetry. NeRFs and 3DGS are also not the only Radiance Field methods that exist. There's quite a lot of them and these won't be all: 𝗥𝗮𝗱𝗦𝗽𝗹𝗮𝘁: https://lnkd.in/eTCRZW42 𝗣𝗹𝗲𝗻𝗼𝘅𝗲𝗹𝘀: https://lnkd.in/eBg8QWAt 𝗧𝗥𝗜𝗣𝗦: https://lnkd.in/ewvQwwHj 𝗤𝘂𝗮𝗱𝗿𝗮𝘁𝘂𝗿𝗲 𝗙𝗶𝗲𝗹𝗱𝘀: https://lnkd.in/ewRJ96YT 𝗦𝗠𝗘𝗥𝗙: https://lnkd.in/edAErJpB 𝗔𝗱𝗮𝗽𝘁𝗶𝘃𝗲 𝗦𝗵𝗲𝗹𝗹𝘀: https://lnkd.in/eVsmC3iF 𝗜𝗡𝗣𝗖: https://lnkd.in/ev_6mtUG 𝟯𝗗 𝗚𝗮𝘂𝘀𝘀𝗶𝗮𝗻 𝗥𝗮𝘆 𝗧𝗿𝗮𝗰𝗶𝗻𝗴: https://lnkd.in/ewkDPmxR 𝗘𝗩𝗘𝗥: https://lnkd.in/e56PAtvX Gaussian Splatting has a head start due to its explicit representation, but what's your favorite Radiance Field method? Will Gaussian Splatting’s hype be eclipsed by Ray Tracing Radiance Fields in six months? Stay tuned—we’re diving into how one major company is using a Radiance Field method next week (hint: it’s neither NeRFs nor Gaussian Splatting!) 👀

111

7 Comments

I recently realized the most valuable aspect of #opensource software to me personally: It's that you can fix problems yourself. When I think back of the unsolved problems I've had over the years, they were all caused by proprietary software. Tech support is a time-consuming frustrating experience, and usually a dead end too. For open-source software I don't need tech support. The answers are all in the code. And I can save others some headache by contributing back. What do you think is the most valuable aspect of open-source software?

9

2 Comments

After reading Andrej Karpathy’s post on X, I’m increasingly convinced that AGI will only be complete when it includes some form of physical systems. Integrating physical components is key to unlocking the full potential of AGI. Check out his thoughts: https://lnkd.in/eRDXsnZT

8

Here's a painful thought... If you see emergent intelligence in LLMs, you might be duped in a way psychics use on their audiences (full argument in linked article) The thing is, it's suprisingly easy to have happen. First, the LLMS use... language. Pretty sophisticated language. We humans tend to associate sophisticated language with intelligence. Just think if the LLM spoke to you like a hillbilly redneck. How would you react? That it sounds like a college educated and reasonable actor prime us to anthropomorphize it much faster than if it were a clucky robot. This helps set the stage for a classic psychic con. People who want to believe there is a consciousness in AI will always find it in today's LLMs. Gary Marcus highlights this essay which is a wonderful follow-on to the one I shared yesterday from Alejandro Piad Morffis. And because I'm proud to be wrangling these problems myself, here's an essay I wrote a few months ago on the risks of anthromoporphizing AI titled "The Biggest Threat from AI is Us! https://lnkd.in/gquF9x_2 https://lnkd.in/gJNGdGXU

8

1 Comment