Chris Shols

100,000 #H100 Clusters: Power, Network Topology, Ethernet vs InfiniBand, Reliability, Failures, Checkpointing 💡Frontier Model Scaling Challenges and Requirements, Fault Recovery through Memory Reconstruction, Rack Layouts There is a camp that feels #AI capabilities have stagnated ever since GPT-4’s release. This is generally true, but only because no one has been able to massively increase the amount of compute dedicated to a single model. Every model that has been released is roughly GPT-4 level (~2e25 FLOP of training compute). This is because the training compute dedicated to these models have also been roughly the same level. In the case of Google’s Gemini Ultra, NVIDIA Nemotron 340B, and Meta LLAMA 3 405B, the FLOPS dedicated were of similar magnitude or even higher when compared to GPT-4, but an inferior architecture was utilized, resulting in these models falling short of unlocking new capabilities. While OpenAI has gotten more #compute, they have mostly directed it at bringing smaller, overtrained, cheaper to inference models such as GPT-4 Turbo and GPT-4o. OpenAI admits they only just started training the next tier of model recently. The obvious next step for #AI is to train a multi-trillion parameter multimodal transformer with massive amounts of #video, #image, #audio, and text. No one has completed this task yet, but there is a flurry of activity in the race to be first. Multiple large AI labs including but not limited to OpenAI/Microsoft, xAI, and Metaare in a race to build #GPU clusters with over 100,000 GPUs. These individual training clusters cost in excess of $4 billion of server capital expenditures alone, but they are also heavily limited by the lack of #datacenter capacity and power as #GPUs generally need to be co-located for high-speed chip to #chip networking. A 100,000 GPU cluster will require >150MW in datacenter capacity and guzzle down 1.59 terawatt hours in a single year, costing $123.9 million at a standard rate of $0.078/kWh. A very big thank you again to Dylan Patel , Daniel Nishball, CFA and SemiAnalysis for the full article with more background, charts and insights via the link below 💡🙏👇 https://lnkd.in/eM_CBmM8 #semiconductorindustry #semiconductors #semiconductors #technology #tech #innovation #artificialintelligence #foundry #tsmc #datacenter #it #ai #cloudcomputing #computing #computer #hpc

51

2 Comments

At Broadcom, we enable today's Gen AI and deliver breakthrough technologies that will power tomorrow’s AI. A cluster of one million XPUs requires over 10 million high bandwidth links. Our SerDes roadmap provides the right foundation for our high performance XPUs and switches to drive power efficiency and scale for these large clusters. Here is our sixth video of the “Enabling AI” video series. #enablingAI #connectedbybroadcom

691

7 Comments

The Transformative Power of Software-Defined Radio 📡   In the fast-changing tech world 🌐, Software-Defined Radio (SDR) is a game-changer, transforming how we use radio communication. Unlike traditional radios tied to fixed hardware, SDR uses software for its functions, offering amazing flexibility and performance 💪.   What is SDR?   SDR is a new way of radio comms. Key components like mixers and filters, usually in hardware, are now handled by software 🖥️. This brings: - Flexibility: Reconfigure fast for different standards and waveforms. - Better performance: Improved signal processing.   For example 👇: - Traditional radio: Limited by hardware. - SDR: Update software to adapt to various tasks.   How SDR Works   SDR combines software and hardware to process signals. Here's a simple view: 1. Antenna: Gets the signals. 2. RF Tuner: Picks the right frequency and filters out the bad. 3. ADC: Turns analog to digital. 4. DDC: Handles digital signal processing. 5. DSP: Decodes and demodulates, with power amps for better audio. 6. Computer: Runs the controlling software.   Key Components of SDR 1. Radio Transceiver: Converts data. Example: Smartphones switching comm standards. 2. Software: Controls and processes. Example: Reprogram for new protocols. 3. Computer: Stores and runs. Example: Laptop for amateur radio.   Awesome Applications of SDR 1. Radio Astronomy: Detect faint cosmic signals. 2. Military: For warfare and secure comms. 3. Emergency Services: Keep comms during disasters. 4. Commercial Wireless: Monitor spectrum and detect interference. 5. Amateur Radio: Communicate and track.   Why SDR is a Big Deal 1. Flexibility: Reconfigure easily for different jobs. 2. Upgradability: Add new features via software. 3. Cost-effective: Cut the need for multiple devices.   Real Impact - Cellular: Helps with 5G integration. - Military: Adapts comms fast. - Emergency: Maintains comms in crises.   Conclusion   SDR is changing our radio world. Its benefits make it great for many uses. As tech moves on, it'll be even more important 🌟.   Discover more at www.DynamicEngineers.com.  #TechInnovation #SDR #WirelessCommunication #RadioTechnology #CommunicationEvolution #SoftwarePower #FutureTech #DigitalSignal #SignalProcessing #FrequencyControl #MicrowaveJournal #MicrowaveJournalCN #DynamicEngineers #EverythingRF

2

Should Lattice Semiconductor buy Altera? With no product portfolio overlap, both companies can strengthen each other. Intel needs a cash injection, selling Altera would be a great way to do that quickly. For Lattice, this would leap frog them into the next tier. Combined revenue would be ~$2.2B, still smaller than AMD's Xilinx hence SEC should not have an issue with it. Before anyone says a smaller company buying a larger one, remember Avago's acquisition of Broadcom? This marriage would be great for the employees, customers, industry, and both companies.

136

29 Comments

#HBM4 set to land sooner than expected 💡 Suppliers and JEDEC are bringing their HBM4 plans forward as they suspect the forecasted Gen #AI boom means incoming NVIDIA GPUs will need it sooner. High bandwidth memory (#HBM) bypasses #X86 socket-connected #DRAM capacity and bandwidth limits by having #memory dies stacked on top of a logic layer, with interconnecting TSV channels, all linked to a #GPU via an interposer component providing higher bandwidth than X86 sockets. The Joint Electron Device Engineering Council (JEDEC) confirms and issues HBM standards, with the current one being its fifth, HBM3E (extended). JEDEC has said it will issue a completed HBM4 spec by the end of this year – just few months ago the schedule was for a 2025 delivery. Preliminary HBM4 characteristics are emerging already. Thanks again to Chris Mellor and Blocks and Files for the full article with more background and insights via the link below 💡🙏👇 https://lnkd.in/eZp_wyHB #semiconductorindustry #semiconductors #semiconductor #technology #tech #innovation #semiconductormanufacturing #it #chip #chips #ic #it

118

4 Comments

people pondering why we don't have 128bit systems in reality a huge number of "computers" being shipped annually, possibly a majority, isn't even 64 bits vast majority of microcontrollers are 32 bit or less and even in a number of 64 bit applications (which the computer industry went to mainly because 32bit could only address 4GB) where custom buses can be designed, address buses are often 48 or 56 bits (and even those can't be fully utilised, just that the architects over-provision) even in very large compute systems the arithmetic also rarely uses 64 bit directly (float in C is 32 bits; to get to 64 you need to use double; it is useful but relatively infrequently), so data bus width of 64 is useful because more parallel wires allow faster bandwidth for packetized data on Network-on-chip (or fabric), and as far as the data bus width is concerned, that is in fact very often larger than 64 bits ( even if the address space is less than 64 bits) in other words, vast majority of computers aren't really 64bits even today in any meaningful sense. just that people don't know about those computers https://lnkd.in/dhFrhVb2

21

8 Comments

Efficiency is key to enable AI at the edge. In this video our VP of AI, Carlos Morales shows how efficiently and fast Ambiq Apollo5 (Cortex M55 on SPOT) can classify images. Reach out to hear more if you are curious! #edgeAI #CortexM55 #arm #efficiencymatters #subthreshold #SPOT #edgecompute #innovation #semiconductors

13

Our game changing inference performance will enable new applications

25

Tech Thought #1 Innovations Beyond HBM in Memory Technology Exciting advancements are on the horizon beyond HBM in memory technology. HBM3 is poised to elevate bandwidth and capacity for demanding applications, while Memory-Class Storage (MCS) merges memory and storage features for enhanced speed and capacity. Persistent Memory (PMEM) offers a blend of DRAM-like performance and storage durability, propelling innovation in data processing efficiency. Furthermore, breakthroughs in 3D integration, like 3D NAND and 3D DRAM, promise heightened storage density and performance. Neuromorphic memory, inspired by neural networks, and Quantum memory, harnessing quantum states, hold the potential to revolutionize speed and capacity in memory technology. Reconfigurable memory architectures are also set to adapt to diverse workloads, optimizing performance across various applications. These advancements collectively mark a significant leap forward in memory technology, paving the way for a future that goes beyond the capabilities of HBM. #MemoryArchitecture #3DMemories #HBM #SpintronicsAI

22

DARK SYSTEMS: Evidence of the acceleration of "Dark Systems" mounts. We can't power the chips we are creating. Also, we are tackling the problem from the wrong side. We assume power is infinite and that we can build more power stations. We are looking at the demand rather than the cause. 1000W+ chips aren't the solution; they cause the power problem. We need to re-look at how fundamental logic is constructed. Our sophistry is thinking modern logic is efficient. Adding more lanes of a Freeway doesn't solve the Freeway problem. We should also note that efficiency is just half the problem. We must also contend with the Jevon Paradox. #vairecomputing #physicsawarearchitectures #nearzeroenergy #physics #reversiblecomputing #jevonparadox URL: https://lnkd.in/gKj-Xr34

10

2 Comments

Jim Keller, former Intel, AMD, and Tesla engineer, aims to challenge Nvidia's dominance in AI with Tenstorrent's energy-efficient chip design. Tenstorrent's upcoming second-generation AI chip promises superior energy and cost efficiency, presenting a viable alternative to Nvidia's offerings. The innovative design of Tenstorrent's chip features independent decision-making cores, potentially reshaping AI chip technology across diverse applications. #AI #Nvidia #semiconductor #semiconductors #chip #technology #innovation #research #computing Tenstorrent NVIDIA https://lnkd.in/g6a3vAu4

31

Our blog post on the Google keynote at OCP is live now: Collaborating on the compute infrastructure of the future: sustainable silicon to intelligent clouds. +Amber Huffman https://lnkd.in/gytPAz_v #googlecloud #ocp

160

3 Comments

On August 9, 2022, the CHIPS and Science Act was signed into law, seizing a historic opportunity to strengthen technology leadership and global competitiveness. As a result of this major legislative milestone, Micron announced our plans to invest in leading-edge memory manufacturing in the U.S. Two years later, thanks to the grants and credits made possible by the CHIPS and Science Act, Micron is continuing to move toward our significant, long-term investment plans. This past April, Micron received $6.1 billion in CHIPS and Science Act funding to support our planned leading-edge memory manufacturing in Idaho and New York. These funds support our plans to invest ~$50 billion in gross capex for U.S. domestic leading-edge memory manufacturing through 2030. That includes the construction of one leading-edge memory manufacturing fab to be co-located with our existing leading-edge R&D facility in Boise, Idaho, and the construction of two leading-edge memory fabs in Clay, New York. With the help of CHIPS Act funding, our planned state-of-the-art semiconductor manufacturing facilities are expected to create ~75,000 domestic jobs over the next 20-plus years and strengthen U.S. economic and national security, furthering American innovation and competition for years to come. While we’ve accomplished so much in these past two years, there’s still more work to be done. We remain committed to helping build a global hub for the memory that powers all computing today and will power the technologies of tomorrow alongside our partners U.S. Senate Majority Leader Chuck Schumer, Governor Kathy Hochul, Governor Brad Little, County Executive Ryan McMahon, and Mayor Lauren Mclean. I speak for all of us at Micron when I say we are truly thrilled to be part of this new era for U.S. semiconductor manufacturing, ushered in by the CHIPS and Science Act. https://lnkd.in/gaazmJDd

313

5 Comments

HBM (High-Bandwidth Memory), which is based on DRAM products, is a crucial building block for data centers. The following article addresses some complex issues in building such products. https://lnkd.in/gSYcAxuN

2

Join this upcoming AMD w ebinar on June 26th: Designing for Versal Adaptive SoCs with AI Engines using Simulink and Vitis Model Composer. This webinar will show an example design – targeting the programmable logic and AI Engines in a Versal AI Edge device, and features speakers from AMD, Avnet, and MathWorks. #TogetherWeAdvance #AdaptiveComputing #Webinar #Versal #Vitis

13

Return to leadership trend continues - 1) Nov 2023: Lead industry on 1st AIPC industry wide CPU, Meteor Lake 2) Sept 2024: quantum leap in industry performance, battery, performance/watt, graphics and AIPC workloads vs ARM variants and AMD - at IFA last month with our Lunar Lake AIPC processor launch 3) Sept 2024: Amazon and US Government award Intel Foundry multi-billion contracts using Intels 18A process 4) and now - per below, industry-analysts highlighting Intel doubling performance in our latest gen Enterprise AI portfolio in Xeon 6 and Gaudi 3 Bold vision - Four nodes in five years. Momentum Inside!

18

Texas Instruments Wins $4.6 Billion in #ChipsAct Grants, Loans - Funding will support $18 billion in spending in #Texas, #Utah - Biden administration almost finished allocating #chips grants Texas Instruments Inc. is set to receive $1.6 billion in Chips Act grants and $3 billion in loans, the #Bidenadministration announced Friday, marking the latest major award from a program designed to boost #American #semiconductormanufacturing. The funding will help pay for one factory in Utah and two in Texas — projects that will cost about $18 billion through 2029 — the Commerce Department said in a statement. The effort is expected to generate around 2,000 #manufacturing #jobs and thousands more in #construction. In total, #TexasInstruments plans to spend around $40 billion across both states, including an additional two factories in #Sherman, Texas. It’s likely that those will come online after 2030, and the Commerce Department has said it will prioritize projects that will be done by the end of the decade. Thanks again to Mackenzie Hawkins and Bloomberg for the full article with more background and insights via the link below 💡🙏👇 https://lnkd.in/eFiP2W4P #semiconductorindustry #semiconductors #semiconductor #technology #tech #it #chips #it #geopolitics #subsidies #usa #innovation #chip #ic

265

2 Comments

My article on SuperComputing Architecture Shift in EFY's Aug Magazine. "The silicon chipset, which is procured from Intel as of today, is the most crucial component in the high-performance compute server. Everything involved in making a supercomputer like the motherboard, the rack, cooling systems, and the software stack, depends on the architecture and features of the silicon chipset. " Electronics For You #Aum #Supercomputing #HPC

151

4 Comments