Michael Graczyk

Along the same vein, I also want to mention Prof. Rao's (Subbarao Kambhampati) argument that many folks may have misinterpreted LLMs' *approximate retrieval* as reasoning. See https://lnkd.in/e5qeYcdn Also, Prof. Rao's group did rigorous work to show that LLMs cannot plan. See https://lnkd.in/exR8Yb5h https://lnkd.in/eJSS7k-H https://lnkd.in/e62WA7e7 #artificialintelligence #machinelearning #deeplearning 

3

NVIDIA's Jensen said "ChatGPT democratized computing, Llama2 democratized generative AI" ...and now Llama3 is the next level 🔥 It's fascinating to watch Meta's strategic moves. With Llama3, they prioritized training on a massive 15T token dataset to pack all into a lean 70B param model, instead of building a massive model. This allows Llama 3 to match trillion+ parameter models like GPT-4, but at 1/10th the compute, storage and inference costs! 💰 This technique was published by Google DeepMind a few years ago https://lnkd.in/gx7VU8aA Meta is not a dark horse in AI anymore. They might be the top dog. https://lnkd.in/gRRJYuxt #llama2 #llama3 #llm #chatgpt #gpt4 #ai #ml

16

5 Comments

If you want to retrieve a list of physics PhDs in NYC or every SF apartment with a Bay Bridge view, you could ask current retrieval systems (e.g. Google) or current intelligence/ reasoning systems (LLMs), but neither can reliably answer that question. To unlock this kind of knowledge in the world, we need a system that can handle extremely complex retrieval requests. That system just does not exist today. LLMs are good at reasoning, but not complex knowledge retrieval. Why is unlocking the world's knowledge even important? Because knowledge underlies everything in our society – what problems we care about, how we act toward others, which politicians we choose. Every societal malfunction is downstream from bad knowledge. Unlocking the world's knowledge is among the most powerful levers of human progress. Exa (Will Bryk) is working on this mission 🚀 Since I was a kid, I've dreamt of having something that truly makes every single piece of knowledge and information in the world, discoverable. Imagine the potential that unlocks for humanity. (I'm not an investor, but this is THE mission to work on!) - they are hiring.

41

2 Comments

Can you really build a company the same way you build an engineering team? I recently touched upon this on the ML Street podcast with Tim Scarfe but let me break it down further. If you’re Jensen Huang running NVIDIA, the answer is yes.  With one of the flattest structures in comparison to company size in the world, Jensen manages 50+ direct reports.  Jensen’s centralized approach to management works as employees aren’t deemed more powerful because of the information they possess. This creates a culture of transparency and freedom of information that… 1. Shows people how and why their work impacts overall business objectives 2. Motivates employees when they know what kind of impact they’re making 3. Aligns everyone to company objectives, reducing friction 4. Builds trust as leadership is pressured to be candid and accountable 5. Fosters innovation through information gathering and collaboration across the company At CentML, we’re a blend of all of the above, where we believe in honesty, craftsmanship, and collaboration, not only within our engineering teams, but our company as a whole. Although, we were in fact founded by a group of engineers, so perhaps a bit of the engineering team rubs off on how we’ve grown the company after all. #ai #engineering #hiring #company #nvidia #centml #management #collaboration #honesty #craftsmanship

44

1 Comment

What a time to be alive. Yes, LLMs are reaching initial plateaus. But then again, the underlying compute is getting 2-3x cheaper and more energy efficient every year. Our models today, are around 40.000x more compute efficient than comparable numerical models. Already today, we can build with 5M or 10M investment, what used to take 200M+ just in infrastructure before. In one year, we can build the same for 3M. In two years for 1.5M. In three years for 0.75M (or less). In other words, a global, high resolution, high precision world model, that would have taken billions to build just 5 years ago, will be built with a few million of investment in the next 5 years. This is our exponential journey towards climate sentience.

48

4 Comments

Let's geek out on OSS vs. closed LLMs for a minute: The #AIatMeta team from Meta just released Chameleon, an open-source multi-modal LLM, sharing "Chameleon is a family of early-fusion token-based mixed-modal models, showcasing broad and general capabilities across various tasks involving images and text." So what does that mean for a business person, exactly? Let's ask the Storytell Chrome extension to de-mystify it for us. Here's the prompt: "I'm a business person who's AI curious but not an LLM expert. I want to use AI to help me and my teams be more productive at work. Explain to me how Chameleon could help do that using non-technical language that my business team can understand." If you want to try the prompt yourself: Here's the technical paper on Chameleon https://lnkd.in/esDdQK3G and you can get the Chrome extension here: https://lnkd.in/ditHm-bd Here's the response to the prompt from Storytell: Chameleon is a cutting-edge AI model that excels at understanding and generating both images and text in a seamless manner. It has been specifically designed to combine visual and textual information effectively, making it highly versatile for a wide range of tasks such as visual question answering, image captioning, text generation, and more. By leveraging Chameleon in your business operations, you can enhance productivity and efficiency in various ways. For example, Chameleon can assist in automating tasks that involve analyzing and generating content based on both images and text. This can streamline processes such as content creation, data analysis, and decision-making by providing valuable insights and generating accurate outputs. Furthermore, Chameleon's ability to outperform other models in text-only tasks and compete with top-performing models in mixed-modal tasks means that it can deliver high-quality results across different types of projects. This can lead to improved accuracy, faster turnaround times, and enhanced overall performance for your business teams. In summary, incorporating Chameleon into your AI toolkit can empower your teams to work more efficiently, make better-informed decisions, and achieve greater productivity in their day-to-day tasks.

9

🚀 Breaking News: Cerebras Systems Inference Gets 3x Faster! Llama 3.1-70B at 2,100 tokens / second⚡️  Here's what this means:  ⚡️ 16x faster than the fastest GPU solution  💪 8x faster than GPUs running Llama3.1-3B (a model 23x smaller!)  🎯 Equivalent to a full GPU generation upgrade (H100/A100) in one software release 🌐Real-World Applications Already Transforming Industries: 🧬 GSK (quote from Kim Branson, SVP of AI/ML): "Cerebras' inference speed is fundamentally improving our drug discovery process through innovative AI applications and intelligent research agents." 🎙️ LiveKit (powering ChatGPT's voice mode): "With Cerebras Inference, what was once the slowest stage in the pipeline is now the fastest" - Russ d'Sa, CEO 🛠️ Technical Improvements Under the Hood: Rewritten & optimized critical kernels (MatMul, reduce/broadcast, element-wise ops) Asynchronous wafer IO computation Advanced speculative decoding implementation Maintained 16-bit original weights precision Verified accuracy by Artificial Analysis ⚡️ What This Means for AI Development: Instantaneous voice & video processing Advanced reasoning capabilities Real-time intelligent applications 10x more reasoning steps without latency increase Enhanced multi-step agentic workflows  Experience the Speed Yourself: Try Cerebras Inference via chat or API at inference.cerebras.ai #LLM #Inference #GenAI

73

1 Comment

If we're only focusing on today's AI costs, we're missing the bigger picture. NVIDIA's Blackwell platform is set to break barriers, potentially cutting inference costs by 25x starting in early 2025. I'm super excited — tons of things to try and can't sleep! LLM + ViT + RAG + LangChain + Tr-VAD combined can be used to build a lot of 0 to 1 products. Where most of the founders are solving for the 1 to n. The future is closer than we think! #predicting

3

Yesterday Nvidia launched Fugatto - a state-of-the-art foundational model for speech. It's a single model, trained on text-to-speech, text-to-music, speech modification, denoising and more. But regardless of the model architecture, you know what's interesting? More than 50% of the paper is dedicated to talking about how they generated datasets for training the model. They use libraries like pedalboard (by Spotify) to generate synthetically alternated samples - variations in speed, timbre, pitch, loudness etc. - and the model then learns to do things like "Make this audio faster". The moat in building multi-modal models will not lie in the architecture. It lies in the data sourcing and data pipelines. Whoever owns the largest source of speech, video data can develop the best models. This comes as a surprise to everyone who keeps asking us "What model do you use at smallest.ai?" Our answer usually is "How does it matter?" It's the data that matters!

466

21 Comments

Optillm: An optimizing inference proxy for LLMs Optillm is an optimizing inference proxy that has over a dozen techniques that aim to improve the accuracy of the responses using test-time compute. Over the last couple of months we have set several SOTA results using smaller and less capable models like gpt-4o-mini. Recently, we have added support for plugins that enable capabilities like memory, privacy and code execution to optillm. Plugins are just python scripts that you can also write yourself, optillm would then load them at start from the directory. You can now also combine the plugins and techniques using & and | operators. E.g. We recently evaluated the new FRAMES benchmark from Google. Using a combination of plugins and techniques (we used readurls&memory-gpt-4o-mini) we were able to get 65.7% accuracy on the benchmark which is very close to what Google reported in their paper with Gemini Flash 1.5 (66.5) which has a context length that is almost 10 times that of gpt-4o-mini. Do check out Optillm at https://lnkd.in/gN6_kNky

21

1 Comment

Wake up - new scaling laws paper just dropped! "Scaling Laws for Precision" Multiple interesting findings: * Training larger models in lower precision can be compute optimal - i.e. for a given compute budget you want to train larger models in lower precision rather than smaller models in higher precision in order to achieve the optimal loss * Too much pretraining data can actually be harmful as you quantize such models to lower precision for inference. Llama models popularized the data saturation regime (training on many more tokens / param than recommended by Chinchilla scaling laws (20 tok/param)) as their goal was not to train a compute optimal model but instead to optimize for inference. These models are more suceptible to degradations due to aggressive quantization. * When number of params is held fixed, compute optimal precision increases approximately logarithmically with data, i.e. for larger models you probably don't want to train in low precision even if your GPUs/training stack supports it! We're hitting the quantization limits that have historically brought us a lot of speed improvements (going from fp32 -> bf16 -> fp8). Lots more details in the paper: https://lnkd.in/dC3_EVrs --- 🔔 If you're looking to hire best ML/tech talent check out my platform here: https://lnkd.in/duJZ3D9T post a job here: https://lnkd.in/df-Ukm8E

169

4 Comments

AI memory is fundamental for conversational AI and AI agents. It is critical for building personalized experiences and improving reasoning. Check out Letta’s course to understand how to use AI memory to enhance the customization and quality of your AI app.

15

$2 H100s: How the GPU Bubble Burst Last year, H100s were $8/hr if you could get them. Today, there's 7 different resale markets selling them under $2. What happened? The market has flipped from shortage ($8/hr) to oversupplied ($2/hr), because of reserved compute resales, open model finetuning, and decline in new foundation model co’s. Rent instead. https://lnkd.in/gpQEidmC

2

1 Comment

Groq, a semi-conductor startup that makes chips for AI inference, was recently valued at $2.8 billion. It’s a classic “overnight success that was years in the making” tale. On the next episode of The Eric Ries Show, I talk with founder and CEO Jonathan Ross, who began the work that eventually led to Groq as an engineer at Google. He was a member of the rapid eval team – “that comes up with all the crazy ideas at Google X.” Nevertheless, he felt the risk involved in leaving to launch Groq in 2016 was far less than the risk of staying in-house and watching the project die. It’s just one of Jonathan’s many counterintuitive beliefs that have paid off. Groq has had many “near-death” experiences in its eight years of existence, all of which have ultimately put it in a much stronger position to achieve its mission: preserving human agency in the age of AI. Jonathan believes that in order to meet it, “[Groq] needs to be in the conversation. The best way to do that is to produce most of the world's AI compute. We want to make sure that there's a lot of voices, and that there isn't control [by] a small number of entities.” Groq is committed to giving everyone access to relatively low cost generative AI compute, driving the price down even as they continue to increase speed. We talked about how the company culture supports that mission, what it’s like to be on the same playing field as companies like Nvidia (and whether it even matters), and Jonathan’s belief that true disruption isn’t just doing things other people can’t do or don’t want to do, but doing things other people don’t believe can be done – even when you show them evidence to the contrary. Other topics we touched on include: 🔵 Why the ability to customize on demand makes generative AI different  🔵 Managing your own and other people’s fear as a founder 🔵 The problems of corporate innovation 🔵 The role of luck in business 🔵 The error of insisting on founder control 🔵 How he thinks about long-term goals and growth 🔵 The importance of empathy 🔵 Maintaining trustworthiness You can listen to or watch the episode here: 🎥 YouTube: https://lnkd.in/gVAM58Vh 🎙️ Spotify: https://lnkd.in/gjuYysrh 🎙️ Apple: https://lnkd.in/gjN459cB If you'd like to read about my main takeaways from the episode, they can all be found here https://lnkd.in/gACeYkuj As always, a big thank you to our sponsors. The show would not be possible without them! ⭐Runway ⭐DigitalOcean ⭐Mercury

411

10 Comments

Going through the list of papers for #ICML2024 and #ICLR2024 and found some interesting ones that I'll share over the next few days. But here's the big one that got me hooked: Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs Paper: https://lnkd.in/dWkn_d_6 Authors: Yeonhong Park and others. Essentially, the paper proposes: ➡️ Rapid generation of multiple models at different quantization levels → Think about having 2, 3, 4, 5, 6, 7 or 8-bit models always in memory, but with a memory requirement of only an 8-bit model (well, almost, they do need some metadata but its an order of magnitude lower memory requirement). ➡️ Specialized LLM serving engine with the support to serve different quantization levels of the same model. ❓ Why is this important ❓ Imagine the scenario where you have a highly finetuned LLM (or an SLM) that you deploy for your users. As a failsafe, you also need to deploy a secondary LLM that comes into play if the first one goes down. However, as soon as the secondary LLM comes online, the users will notice - it's not a finetuned version of your first model! Or even if you deploy a finetuned version, you end up deploying TWO models, in memory, in parallel. This paper solves this problem - a possible failsafe method for an in-production LLM that failovers to smaller/higher quantization levels based on bandwidth and user requirements. Incredible! For a truly redundant system though this still doesn't work because if the LLM serving engine goes down, all models go down as well. So don't throw away your failsafe LLM just yet, but this might be a handy alternative. Book a free 15 minute call with me: https://lnkd.in/dGnHs_Mn #icml2024 #iclr2024 #llm #nlp #machinelearning #consulting #generativeai #openai #microsoft #iclr #icml #research #paper

12

Yann LeCun explains that Meta's internal codename for AGI is AMI, which stands for Advanced Machine Intelligence. The word is pronounced like the French word for friend. 🙄 1. AMI is yet another name for AGI; like AGI, it has no actual definition. As I have mentioned before, the number of categories and names used to explain an area of research is inversely proportional to the available knowledge about that research. As more companies create AGI strategies, the number of names will increase. 2. LeCun says, "We need machines that resemble human intelligence," but later says that "he doesn't like AGI (thus preferring "AMI") because human intelligence isn't general at all. It's specialized." He wants machines to resemble human intelligence by not resembling human intelligence. 🤷‍♂️ 3. LeCun says we need world models so machines can understand like humans. Sadly, humans don’t have an internal model of the exterior world. This argument is just a recapitulation of the homunculus argument. If there were a "world model," who would interpret it? Therefore, a world model for a machine would be uniquely computational, and while it might positively impact performance, it would have no impact on "understanding." 4. What bothers me most about this nonsense is the depiction of humans as specialized, even while saying machines must navigate the world with understanding and reasoning, like humans. This is a contradiction, treating humans in ways they do not deserve and creating a false impression that human potential is inert. This attitude is very different from that of researchers from the '60s and '70s, who exaggerated AI's potential but did not find it necessary to do so at the expense of humans. Science fiction writer Robert Heinlein’s “competent man” was often seen as the idealized archetype for AI researchers.[1] Heinlein writes that “A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly.”[2] Heinlein adds, “Specialization is for insects.” While most of us can’t do half of these things, there’s a truth in Heinlein’s vision: human potential is real and expansive and more than a collection of specialized skills.

65

34 Comments

Nearly a decade ago, I wrote out a complete and rigorous proof of Cantor's theorem that the set of sequences of natural numbers is bigger than the set of natural numbers...using only words of one syllable. https://lnkd.in/g4wWYVYc This is not the only time I revisited the result, much more recently I formally published a number of generalisations of this using different fundamental assumptions (for experts: taking Lawvere's fixed-point theorem and the contrapositive diagonal argument and showing how the proof works in a substructural type of logic). But this one-syllable-word proof was so much fun. I was reminded of this because someone got in touch with me, saying how it is something he periodically re-read for pure interest's sake, and asking if I could recover some material that was on the now-vanished Google+ to which I linked. I had in fact saved that discussion, which had contributions of similar pieces from a number of other mathematicians, including the Fields medal winner Tim Gowers. I will put a pdf of that thread somewhere public, soon, and link to it. But my correspondent was so taken by the exercise that he said he'd go so far as to purchase a physical book containing a collection of such "short words proofs", if one existed. The idea of publishing a slim volume of fun little proofs certainly is appealing, in some respects, because I love producing physical artifacts that reflect mathematics, but the market must surely be tiny!

7

2 Comments

Much has been said about many companies’ desire for more compute (and data) to train large foundation models. I think it’s under-appreciated that we also have nowhere near enough compute available for inference on foundation models. Years ago, when I was leading teams at Google, Baidu, and Stanford that focused on scaling up deep learning, many semiconductor makers, data center operators, and researchers asked me if AI would continue to make good use of more compute if they kept delivering it. For many desktop workloads, like running a web browser, a faster CPU doesn’t help much beyond a certain point. So do we really need faster and faster AI processors? Each time, I confidently replied “yes!” and encouraged them to keep scaling up compute. (Sometimes, I added half-jokingly that I had never met a machine learning engineer who felt like they had enough compute. 😀) Fortunately, this prediction has been right so far. However, beyond training, we are also far from exhausting the benefits of faster and higher volumes of inference. Today, a lot of LLM output is for human consumption. A human might read around 250 words per minute, which is around 6 tokens per second (250 words/min / (0.75 words/token) / (60 secs/min)). So it might seem there’s little value to generating tokens much faster than this. But in an agentic workflow, an LLM might be prompted repeatedly to reflect on and improve its output, use tools, plan and execute multiple steps, or implement multiple agents that collaborate. So, we might generate hundreds of thousands of tokens or more before showing any output to a user. This makes fast token generation very desirable and makes slower generation a bottleneck to taking better advantage of existing models. That’s why I’m excited about the work of companies like Groq, which can generate hundreds of tokens per second. Recently, SambaNova also showed it can hit hundreds of tokens per second. Incidentally, faster, cheaper token generation will also help make running evaluations (evals), which can be slow and expensive today since it involves iterating over many examples, more palatable. Fortunately, both training and inference are rapidly becoming cheaper. I spoke with Cathie Wood and Charles Roberts of the investment firm ARK, which is famous for its bullish predictions on tech. They estimate that AI training costs are falling 75% a year. If they are right, a foundation model that costs $100M to train this year might cost $25M to train next year. Further, they report that for “enterprise scale use cases, inference costs seem to be falling at an annual rate of ~86%, even faster than training costs.” I don’t know how accurate these specific predictions will turn out to be, but with progress in both semiconductors and algorithms, I do see training and inference costs falling rapidly. This will be good for application builders and help AI agentic workflows lift off! [Original text: https://lnkd.in/dJ9tVGh7 ]

2,899

80 Comments

Solving coding is key to artificial general intelligence. This is why ALL the leading AI labs and companies work with Turing to help LLMs get better at coding, reasoning and thinking. Zuck: “For Llama 3, we focused on training with a lot of code Training the model on code helps it reason across a lot of different types of domains If someone else solves reasoning, and we're sitting here with a basic chatbot, our product is lame We realized we've got to solve general intelligence And we upped the investment to make sure that we could do that” https://lnkd.in/eY66Gpf3

156

7 Comments