Yann LeCun’s Post

Llama 3 8B and Llama 3 70B — trained on two custom-built 24,000 GPU clusters — are are among the best-performing generative AI models available today. Llama 3 8B bests other open source models like Mistral’s Mistral 7B and Google’s Gemma 7B, both of which contain 7 billion parameters, on at least nine benchmarks: MMLU, ARC, DROP, GPQA (a set of biology-, physics- and chemistry-related questions), HumanEval (a code generation test), GSM-8K (math word problems), MATH (another mathematics benchmark), AGIEval (a problem-solving test set) and BIG-Bench Hard (a commonsense reasoning evaluation).

Llama3 is out! I'm particularly excited to learn more about LlamaGuard2 and the other guardrails like CyberSecEval2 (https://lnkd.in/eSabvvsv). I also love seeing ecosystems come together. Llamaguard2 and the risk taxonomy used by MLCommons are the same. This interoperability is critical. Taxonomies will inevitably change over time (perhaps created by a broader participatory process), and building software in a way that it can take advantage of the best ecosystem learnings is critical. Of course, I'm thinking of Credo AI as well. We actively explore open-source models and guardrails for our own use and our customers. We benefit from (and contribute to!) our distributed responsible AI intelligence. Governance is a way to systematize best practices and oversight - and I love more tools to build on top of.

🥁 Exciting News in the AI Sphere! 🥁 Llama3 is officially out! Today marks the release of both 8B and 70B models of Llama3, promising a substantial leap forward in AI capabilities. With an impressive 8k context length and trained on a massive 15 trillion tokens, Llama3 represents a significant technological advancement.

Meta 's relentless pursuit of "compute" ... by "the end of 2024, we’re aiming to continue to grow our infrastructure ... 350,000 NVIDIA H100 GPUs". Total Power Consumption = 350,000 * 700W = 245 MW For every 2W, you need 1W for HVAC so est total 370 MW. That is equivalent energy to power the whole city of San Francisco. src: https://lnkd.in/gUP49eFZ Supplemental: The average electricity customer in San Francisco, CA uses 729 kWh of electricity per month, and 8,748 kWh over the course of the year. Given that there are approximately 361,222 households in San Francisco, we can calculate the total power consumption for the city 361 Megawatts of power.

Excited to try this out - especially with integration in Meta AI chatbot. We will see if it’s better than Claude Opus or ChatGPT-4 at python. Waiting for the benchmarks from dev experts! EDIT: a quick test of Meta AI, shows it’s still Llama 2. Llama 3 is only downloadable right now

Very exciting we get an on-prem model that can do basic agentic workflows. It’s still a bit less powerful at reasoning than GPT-4-Turbo, which is less powerful than GPT-4. That reasoning gap matters a lot for complex agentic workflows, e.g. agent swarms, agent graphs, agent hierarchies, and Society of Mind emulation.

18 April : 3 interesting tidbits : 1. A big change in Llama 3 compared to Llama 2 is the use of a new tokenizer that expands the vocabulary size to 128,256 (from 32K tokens in the previous version). This larger vocabulary can encode text more efficiently (both for input and output) and potentially yield stronger multilingualism. 2.The Llama 3 models were trained ~8x more data on over 15 trillion tokens on a new mix of publicly available online data on two clusters with 24,000 GPUs. 3. 10 Million human-annotated data samples with combination of supervised fine-tuning (SFT), rejection sampling, proximal policy optimization (PPO), and direct policy optimization (DPO). https://lnkd.in/gh_Vdu3n

We are moving fast towards high performing small language models which would be the real deal in my opinion as the much smaller size will enable their usage on device and will open the door for realtime usecases to leverage their power.

Another day, yet another new model: Meta releases Llama3. We've barely finished trying out Mistral AI’s latest LLM, and now we have another one in the queue. The exploratory space for AI engineers keeps expanding. The positive aspect, however, is the commoditization of LLMs: they're becoming faster, better, and cheaper, with support for tool integrations enabling independent agents. The world is your oyster. I'm uncertain if, ten years from now, we'll see as many companies producing foundational models. I envision a future with only a few powerful foundational models, akin to the evolution of the cloud computing space (dominated by three major players) or chip design space (dominated by a few giants). Yet, there will be thousands of companies developing and selling apps built on top of these super powerful models, after customizing and fine-tuning them to their specific domains. Until then, happy exploring in the LLM world. #meta #ai #llm #generativeai

🎊Meta's smallest LLM released today (Llama3 8b) is now as powerful as ~9x bigger model from it's previous generation (Llama2 70b). But why did they decide to go with 8b parameters when all other models in the category are 7b? 🤔 ie. Mistral 7b, Gemma7b, WizardLM 7b My guess is this: Meta probably didn't want to spend a few months training a new 7b model... ... and then find out that it underperformed an existing model in the 7b category. 🤦 A quick solution: make it a bit bigger. Bigger is better. But not so big that it's close to the 13b category. 😉 However, this is still a HUGE STEP forward! This could also mean current 7b models could be squeezed to be even smaller. 🍊 We'll probably soon see 2b models with the equivalent of today's 7b models. Let's see how the open source community takes this forward. Meta is one of the only companies that's putting out true open source work worth millions. And this is a fantastic addition. Truly amazing times to live in! ✨ #GenAI #llm #llama #meta