Ronald Ding’s Post

Chinese GPU maker Moore Threads' MTLink fabric tech challenges Nvidia's NVLink, can now scale to 10,000 GPUs for AI clusters and 1280000 Tensor cores. More GPUs, more performance. Moore Threads has upgraded its KUAE data center server for AI, enabling connecting up to 10,000 GPUs in a single cluster. The KUAE data center servers integrate eight MTT S4000 GPUs interconnected using the proprietary MTLink technology designed specifically for training and running large language models (LLMs). These GPUs are based on the MUSA architecture and feature 128 tensor cores and 48 GB GDDR6 memory with 768 GB/s of bandwidth. A 10,000-GPU cluster wields 1,280,000 tensor cores, but the actual performance is unknown as performance scaling depends on numerous factors. Moore Threads recently completed a financing round, raising up to 2.5 billion yuan (approximately US$343.7 million). This influx of funds is expected to support its ambitious expansion plans and technology advancements. However, without access to advanced process technologies offered by TSMC, Intel Foundry, and Samsung Foundry, the firm faces numerous challenges on the path to developing next-gen GPUs.

Nvidia has just unveiled the Blackwell B200 GPU and GB200 "superchip," which they claim to be the most powerful chip for AI. 💪 I'm thrilled to see the advancements Nvidia is making in this field. The B200 GPU boasts an impressive 208 billion transistors and can deliver up to 20 petaflops of FP4 performance. 🤯 But what really caught my attention is the GB200, which combines two B200 GPUs with a single Grace CPU. Nvidia claims this configuration can provide 30 times the performance for LLM inference workloads while consuming up to 25 times less energy compared to the H100. 🔋 This is a game-changer for AI training and inference. Nvidia CEO mentioned that training a 1.8 trillion parameter model, which previously required 8,000 Hopper GPUs and 15 megawatts of power, can now be done with just 2,000 Blackwell GPUs consuming only four megawatts. 🌍 The second-gen transformer engine and the next-gen NVLink switch are the key improvements that make this possible. The transformer engine doubles the compute, bandwidth, and model size by using four bits for each neuron, while the NVLink switch allows 576 GPUs to communicate with each other at an incredible 1.8 terabytes per second of bidirectional bandwidth. 🚀 It's exciting to see major cloud service providers like Amazon, Google, Microsoft, and Oracle already planning to offer the NVL72 racks, which combine 36 CPUs and 72 GPUs in a single liquid-cooled rack for a total of 720 petaflops of AI training performance or 1,440 petaflops of inference. ☁️ As an AI professional, I can't wait to see how these advancements will accelerate the development and deployment of AI solutions across various industries. Nvidia's Blackwell architecture is set to revolutionize the field of AI, and I'm eager to explore its potential in my own work. 🔍 What are your thoughts on Nvidia's latest announcement? Let me know in the comments below! 💬 #Nvidia #ArtificialIntelligence #Blackwell #B200 #GB200 #AIChips #MachineLearning #DeepLearning

Nvidia Reveals Blackwell B200 GPU, the 'World's Most Powerful Chip' For AI: Sean Hollister reports via The Verge: Nvidia's must-have H100 AI chip made it a multitrillion-dollar company, one that may be worth more than Alphabet and Amazon, and competitors have been fighting to catch up. But perhaps Nvidia is about to extend its lead -- with the new Blackwell B200 GPU and GB200 "superchip." Nvidia says the new B200 GPU offers up to 20 petaflops of FP4 horsepower from its 208 billion transistors and that a GB200 that combines two of those GPUs with a single Grace CPU can offer 30 times the performance for LLM inference workloads while also potentially being substantially more efficient. It "reduces cost and energy consumption by up to 25x" over an H100, says Nvidia. Training a 1.8 trillion parameter model would have previously taken 8,000 Hopper GPUs and 15 megawatts of power, Nvidia claims. Today, Nvidia's CEO says 2,000 Blackwell GPUs can do it while consuming just four megawatts. On a GPT-3 LLM benchmark with 175 billion parameters, Nvidia says the GB200 has a somewhat more modest seven times the performance of an H100, and Nvidia says it offers 4x the training speed. Nvidia told journalists one of the key improvements is a second-gen transformer engine that doubles the compute, bandwidth, and model size by using four bits for each neuron instead of eight (thus, the 20 petaflops of FP4 I mentioned earlier). A second key difference only comes when you link up huge numbers of these GPUs: a next-gen NVLink switch that lets 576 GPUs talk to each other, with 1.8 terabytes per second of bidirectional bandwidth. That required Nvidia to build an entire new network switch chip, one with 50 billion transistors and some of its own onboard compute: 3.6 teraflops of FP8, says Nvidia. Further reading: Nvidia in Talks To Acquire AI Infrastructure Platform Run:ai Read more of this story at Slashdot.

Nvidia Reveals Blackwell B200 GPU, the 'World's Most Powerful Chip' For AI: Sean Hollister reports via The Verge: Nvidia's must-have H100 AI chip made it a multitrillion-dollar company, one that may be worth more than Alphabet and Amazon, and competitors have been fighting to catch up. But perhaps Nvidia is about to extend its lead -- with the new Blackwell B200 GPU and GB200 "superchip." Nvidia says the new B200 GPU offers up to 20 petaflops of FP4 horsepower from its 208 billion transistors and that a GB200 that combines two of those GPUs with a single Grace CPU can offer 30 times the performance for LLM inference workloads while also potentially being substantially more efficient. It "reduces cost and energy consumption by up to 25x" over an H100, says Nvidia. Training a 1.8 trillion parameter model would have previously taken 8,000 Hopper GPUs and 15 megawatts of power, Nvidia claims. Today, Nvidia's CEO says 2,000 Blackwell GPUs can do it while consuming just four megawatts. On a GPT-3 LLM benchmark with 175 billion parameters, Nvidia says the GB200 has a somewhat more modest seven times the performance of an H100, and Nvidia says it offers 4x the training speed. Nvidia told journalists one of the key improvements is a second-gen transformer engine that doubles the compute, bandwidth, and model size by using four bits for each neuron instead of eight (thus, the 20 petaflops of FP4 I mentioned earlier). A second key difference only comes when you link up huge numbers of these GPUs: a next-gen NVLink switch that lets 576 GPUs talk to each other, with 1.8 terabytes per second of bidirectional bandwidth. That required Nvidia to build an entire new network switch chip, one with 50 billion transistors and some of its own onboard compute: 3.6 teraflops of FP8, says Nvidia. Further reading: Nvidia in Talks To Acquire AI Infrastructure Platform Run:ai Read more of this story at Slashdot.

Nvidia's next-gen AI GPUs could draw an astounding 1000 Watts each, a 40 percent increase. Dell, one of the world's largest server makers, has spilled the beans on Nvidia's upcoming AI GPUs, codenamed Blackwell. Apparently, these processors will consume up to 1000 Watts, a 40% increase in power over the prior-gen, requiring Dell to use its engineering ingenuity to cool these GPUs down. Dell's comments might also hint at some of the architectural peculiarities of Nvidia's upcoming compute GPUs - https://lnkd.in/gySXzNEu #AI #nvidia

🌟 Major announcements from Nvidia this week... and a significant rise in their stock price 📈 Nvidia's CEO Jensen Huang spoke at COMPUTEX 2024: ➜ Sustainable Computing: Highlighting the power of accelerated computing, Huang shared that the combination of GPUs and CPUs can deliver up to a 100x speedup with only a 3x increase in power consumption, achieving 25x more performance per Watt over CPUs alone. ➜ Innovative Chips on the Horizon: The Blackwell Ultra chip, set for a 2025 release, continues to push the boundaries of AI technology. Following that, the Rubin platform, expected in 2026, will feature new GPUs, a new Arm-based CPU—Vera—and advanced networking capabilities. ➜ NIMS – NVIDIA Inference Microservices: Pre-trained AI models provided as optimized containers, ready for deployment in the cloud or private data centers. Meanwhile Nvidia’s valuation surpassed $3 trillion yesterday making it the third company ever to reach that milestone. 🚀 Nvidia also has a 10-to-1 stock split effective tomorrow, June 7, likely contributing to the recent stock surge as well More info: https://lnkd.in/dAY9zww9 #NVIDIA #COMPUTEX2024 #AI #ArtificialIntelligence

Nvidia’s latest chip promises to boost AI’s speed and energy efficiency. What’s new: The market leader in AI chips announced the B100 and B200 graphics processing units (GPUs) designed to eclipse its in-demand H100 and H200 chips. The company will also offer systems that integrate two, eight, and 72 chips. How it works: The new chips are based on Blackwell, an updated chip architecture specialized for training and inferencing transformer models. Compared to Nvidia’s earlier Hopper architecture, used by H-series chips, Blackwell features hardware and firmware upgrades intended to cut the energy required for model training and inference. Training a 1.8-trillion-parameter model (the estimated size of OpenAI’s GPT-4 and Beijing Academy of Artificial Intelligence’s WuDao) would require 2,000 Blackwell GPUs using 4 megawatts of electricity, compared to 8,000 Hopper GPUs using 15 megawatts, the company said. Blackwell includes a second-generation Transformer Engine. While the first generation used 8 bits to process each neuron in a neural network, the new version can use as few as 4 bits, potentially doubling compute bandwidth. A dedicated engine devoted to reliability, availability, and serviceability monitors the chip to identify potential faults. Nvidia hopes the engine can reduce compute times by minimizing chip downtime. Nvidia doesn’t make it easy to compare the B200 with rival AMD’s top offering, the MI300X. Price and availability: The B200 will cost between $30,000 and $40,000, similar to the going rate for H100s today, Nvidia CEO Jensen Huang told CNBC. Nvidia did not specify when the chip would be available. Google, Amazon, and Microsoft stated intentions to offer Blackwell GPUs to their cloud customers. Behind the news: Demand for the H100 chip has been so intense that the chip has been difficult to find, driving some users to adopt alternatives such as AMD’s MI300X. Moreover, in 2022, the U.S. restricted the export of H100s and other advanced chips to China. The B200 also falls under the ban. Why it matters: Nvidia holds about 80 percent of the market for specialized AI chips. The new chips are primed to enable developers to continue pushing AI’s boundaries, training multi-trillion-parameter models and running more instances at once. We’re thinking: Cathie Wood, author of ARK Invest’s “Big Ideas 2024” report, estimated that training costs are falling at a very rapid 75 percent annually, around half due to algorithmic improvements and half due to compute hardware improvements. Nvidia’s progress paints an optimistic picture of further gains. It also signals the difficulty of trying to use model training to build a moat around a business. It’s not easy to maintain a lead if you spend $100 million on training and next year a competitor can replicate the effort for $25 million. Andrew Ng, [email protected]

NVIDIA has left the tech word speechless yet again... “We need bigger GPUs. So ladies and gentlemen, I would like to introduce you to a very, very big GPU,” is the most recent quote from their CEO. NVIDIA has revealed the Blackwell B200 GPU which is the world's most powerful chip for AI... by far. They are saying that the new GPU offers up to 20 petaflops of FP4 horsepower from its 208 billion transistors and that a GB200 that combines two of those GPUs with a single Grace CPU can offer 30 times the performance for LLM inference workloads. Their share price just does not seem to stop rocketing, it's now up to $1348 ($US884.55), giving it a market capitalisation in the trillions. NVIDIA says the new chip reduces cost and energy consumption by up to 25x compared to its famed H100. It's going to be interesting to see what the generative ai race looks like now. Expect it to ramp up quickly with these new GPU's.