Amir Zarkesh

Our game changing inference performance will enable new applications

25

You've seen those long context Google Gemini demos? Here's us cheekily unlocking Long Context Workflows on AMD MI300X with our Flywheel Inference Engine on TensorWave. No doubt that long context is making some buzz around LLM inference. It is kinda wild that you can just dump your entire doc set into the context and then... just chat with it. No RAG, no finetuning or anything. Just a chat with a looooong prompt.

32

1 Comment

The magic is the speed of the chip to chip connection.

21

2 Comments

Really interesting to see this: how Europe is moving up. It is interest to contrast this qualitative survey with the #siliconcatalyst analysys that UK is sixth in Europe in funding value fir silicon investment. And I will put in a plug for our #UKTIN Forward Capabilities paper on recommendations for UK Semiconductor sector: https://lnkd.in/eNNAsCAz

8

Groq, known for their high-performance AI hardware and software, released two new open-source models specifically designed for tool use: Llama-3-Groq-70B-Tool-Use and Llama-3-Groq-8B-Tool-Use, built with Meta Llama-3. Tools are utilized by AI assistants to respond to user prompts. Typically, a tool is an API call to an external system, like Google search. The results from the tool are then analyzed by the LLM and incorporated into the response. Thus, the LLM's performance in using tools is paramount. These models promise significant advancements in processing speed and efficiency. Llama-3-Groq-70B-Tool-Use boasts 70 billion parameters, while Llama-3-Groq-8B-Tool-Use features 8 billion parameters, both optimized for rapid deployment in various applications. By leveraging Groq’s advanced hardware, these models aim to set a new standard in AI tool use, focusing on enhancing both performance and accessibility.

1

Here’s why Baya Systems made my list of the Top 13 AI/ML Startups to Bet On Changing the World… There’s no shortage of hardware startups trying to succeed in the realm of AI; our team recently counted around 210 new (started in the last 3-5 years) companies in the space. What makes any of them worth betting one’s career on is debatable, but Baya has the recipe figured out because even before they emerged from stealth, they had attracted some of the industry’s TOP talent to join their mission. Sailesh Kumar is one source of ‘gravitational pull’ for Baya’s talent. Before co-founding Baya, he had an illustrious career as Chief Architect at Huawei, founded Netspeed, and was a Fellow an Intel where he pioneered projects like the standardization of data center compute protocols. Those experiences gave him keen insights into the growing demand for compute, catalyzed by the rapid rise of AI, as well as the challenges system architects would face building next-generation AI systems. With the main culprits being the Memory Wall and the complexity of software, there were no apparent answers, and that was his catalyst. Sailesh launched a mission to break down the barriers preventing scalable AI by leveraging chiplets, aiming to usher in what he refers to as “The Chiplet Era” and during those first couple of years, Baya developed in stealth because chiplets were starting to gain traction and he didn’t want to tip their hand to the market. This is also when he did some expert-level recruiting. Tapping Jim Keller to join their Board gave Baya additional credibility and a galactic level of pull. Soon, other key leaders joined. Swapnil Lotlikar came over from a career at “big semi” companies and his expertise in cache coherence, memory, and SoC fabrics wasn’t just additive to Baya, it was exponential. Then Sailesh got a second star into orbit with the hire of @James Aldis, whose expertise in SoC interfaces and fabrics has defined standards, resulting in an industry ecosystem of IP that is critical to the evolution chiplets. Sailesh then hired Ameen Ashraf to be a Fellow and attracted Nandan Nayampally and Ravikanth Reddy into their planetary alignment. Nandan helped grow ARM’s CPU business to $1 billion and oversaw some of the world’s first deployments of neuromorphic Edge AI chips as CMO at Brainchip, and Ravi has been instrumental in developing interconnect fabrics like CXL and others that have set the stage for advanced interconnects. Building a star-studded team while in stealth speaks volumes about a leader and in my experience with working with founders to build their core teams, the quality of that team is the best gauge for measuring a startup’s chance of survival. Sailesh’s next challenge is execution and my advice to him and Baya is: don’t lose focus on your technical goals as a result of talent identification and attraction. Delivering WHILE scaling is much tougher. #semiconductorindustry #artificialintelligence #chiplets

62

These are the top alternative chips to NVIDIA GPUs that we’re excited about SambaNova Systems SN40L RDUs are inference oriented chips leveraging 3 tiers of memory to optimize for running larger parameter models on less hardware. There is software optimization to ensure the most frequently used model weights are the fastest memory while less frequently used model weights are on the slower memory. Groq LPUs are inference oriented chips that focus on deterministic, pipeline processing vs the massively parallel processing of GPUs. Each chip has 256 MB of the fastest SRAM memory which requires a large number chips to run models. However, direct chip to chip connections makes the transfer speeds incredibly fast. Cerebras Systems WSE-3 chips are physically larger chips compared to GPUs that allow for more transistors, cores, and on-chip memory. It emphasizes the ability to process more on a single chip without having to scale out horizontally to multiple chips which can help engineers save time and effort in optimizing for distributed processing. Intel Corporation Gaudi 3 chips are designed for general purpose AI including training, fine tuning, and inference. As a direct competitor to Nvidia’s H100, the Gaudi 3 performs slightly better across the board in benchmarks and tests than the H100s. AMD MI300x chips are AMD’s response to Nvidia’s dominance in AI oriented GPUs. It is better than H100s across the board on hardware specifications. A key feature of the MI300x is its 192 GB of memory which allows for much larger models, such as the massive 403B Llama 3, to fit on a single node of 8 chips. Despite these chips seemingly beating Nvidia’s Hopper generation of chips on hardware specs, Nvidia’s chips are still the go to for most data center deployments and workloads. The key difference is Nvidia’s CUDA software for optimization that allows Nvidia chips to perform on par or better than chips that have better specs on paper. In addition to the optimization, CUDA is also deeply ingrained in the developer ecosystem and is always natively supported for new projects that enable an ecosystem lead.  This is all ignoring the fact that Nvidia's upcoming Blackwell generation of chips are supposed to significantly improve upon current gen chips. What will the chip landscape look like in 5 years?

80

15 Comments

Jensen Huang co-founded NVIDIA in 1993 with Chris Malachowsky and Curtis Priem, driven by a vision to revolutionize computer graphics. At the time, Huang was 30 years old and brought valuable experience from his work at LSI Logic and AMD. NVIDIA started in a small office in Santa Clara, California, focusing on developing graphics processing units (GPUs) to enhance gaming experiences. Their first major product, the NV1, launched in 1995, combined 2D and 3D graphics with sound. However, it faced limited success due to its unconventional design. Despite early challenges, NVIDIA persevered and gained traction with the release of the RIVA series in 1997, marking its entry into the competitive PC gaming market. Under Huang’s leadership, NVIDIA continued to innovate, eventually dominating the GPU industry and expanding into fields like artificial intelligence, data centers, and autonomous vehicles, becoming a global leader in computing technology. #jensenhuang #nvidia #enterpreneurs #entrepreneurlife #success #mindset #entrepreneur #enterpreneurship #business #motivation

532

14 Comments

Are Google's Tensor Processing Units in-house and proprietary? Technically, yes. But with Instachip, you can pip install parts of an open-source TPU into your own hardware project! Before that—what is Instachip? It's a tool that lets you pip install Verilog modules into your hardware designs. From the command line, type "vpm include <COMPONENT>", and Instachip will download the component and provide detailed documentation, pinouts, waveforms, etc. Our users have used it to install hardware as complex as a Tensor Processing Unit in a single command! Hobbyists, students, and NVIDIA engineers all use Instachip, so it's quite reliable (most users use it offline, so no data concerns). Search for any module to include: https://lnkd.in/eJyYgUCK Download Instachip now: https://lnkd.in/e_RTRDDD Before you download—watch how YOU can include Evan Lin's open-source TPU core in your own project 👇

317

7 Comments

5 Under-the-Radar AI Hardware Companies that Potentially can Dethrone Nvidia While Nvidia dominates the AI hardware scene, several exciting companies are pushing boundaries and could emerge as strong competitors. 🚀 Nvidia: Focus: Versatile Graphics Processing Units (GPUs) Strength: Powerful and general-purpose. Their GPUs excel at various tasks beyond AI, including graphics processing, scientific computing, and video editing. This makes them a widely adopted solution across different industries. Weakness: While powerful, Nvidia's GPUs may not be the most efficient for specific AI tasks compared to specialized hardware. Additionally, their high power consumption might not be suitable for resource-constrained environments. 🚀 Key Difference: Nvidia offers a one-size-fits-all solution, whereas these emerging companies focus on specific areas within AI hardware: Here are five to keep an eye on: 🚀 Groq Focus: High-performance processors for low-latency AI applications. Moat: Unique chip architecture designed from the ground up for AI, offering near-linear scaling and efficient resource utilization. This focus on a specific application area allows for superior performance in low-latency tasks compared to general-purpose GPUs. 🚀 Hailo Focus: Energy-efficient Deep Learning processors for edge devices (like robots and autonomous vehicles). Moat: Hailo chips are specifically designed for smaller form factors with lower power requirements. This makes them ideal for applications where size and power consumption are critical, potentially disrupting the market for on-device AI processing. 🚀 Mythic Focus: Analog AI processors that use less power and offer higher performance for specific AI tasks. Moat: Mythic leverages analog computing, a different approach compared to traditional digital processors. This allows for potentially lower power consumption and faster processing for specific AI algorithms, though the technology is still relatively new and needs wider adoption. 🚀 SambaNova Systems Focus: Hardware and software co-design for large-scale AI training and inference. Moat: SambaNova offers a complete AI stack, including their own hardware and software designed to work together seamlessly. This tight integration can potentially improve performance and efficiency compared to using separate hardware and software components. 🚀 Cerebras Systems Focus: Custom-designed AI accelerators for supercomputers, tackling massive AI workloads. Moat: Cerebras builds some of the largest AI chips in the world, specifically designed for training complex AI models on massive datasets. This caters to a niche market segment but offers unparalleled processing power for specific applications. Who else do you think can compete and win over Nvidia. A detailed post is in comments. for all semiconductor and AI related content, Follow TechoVedas

356

29 Comments

Level up Chip Design Innovation with MATLAB and Silicon Catalyst! #MathWorks News & Stories brings you an exciting collaboration between MathWorks and Silicon Catalyst. This article explores how MATLAB an Simulink empowers chip-based semiconductor startups to develop cutting-edge solutions in medicine and wireless technology. Key Takeaways: >> Richard Curtin, Managing Partner at Silicon Catalyst, highlights the growing demand for "specialized design tools to overcome the complexities of modern chip design." >> Raphael Guimond, Antenna Designer at SPARK Microsystems, emphasizes how MATLAB's capabilities streamline the antenna design process, allowing engineers to "focus on innovation rather than repetitive tasks." ➡️ Dive deeper and discover how MATLAB can: >> Shorten design cycles through its powerful simulation and modeling tools. ⏱️➡️ >> Optimize performance with efficient algorithms and code generation capabilities. >> Facilitate collaboration with a robust platform for data sharing and version control. Don't miss out! Read the full story here: https://lnkd.in/eAv7JhYV #engineering #semiconductor #wireless #innovation #MATLAB #medicine #Simulink #Probius #SPARK #Microsystems #SiliconCatalyst

4

MSR has showcased BitNet, a novel 1-bit Transformer architecture, designed to address the challenges associated with scaling LLMs. A key innovation in BitNet is the introduction of BitLinear, which replaces traditional linear layers and facilitates the training of 1-bit weights from the ground up. This architecture not only achieves competitive performance in language modeling but also significantly reduces memory usage and energy consumption when compared to advanced 8-bit quantization methods and FP16 Transformer baselines.   The architecture employs quantization-aware training, optimizing the model for low-precision representation from the outset, which enhances both accuracy and stability during the training process. BitNet features stacked blocks of self-attention and feed-forward networks with binarized weights, while retaining higher precision for other components, thereby minimizing computational costs. This design allows for efficient matrix multiplication with 1-bit weights, resulting in notable energy savings, especially as model sizes increase.   The training methodology incorporates techniques such as the straight-through estimator for gradient approximation and mixed precision training. These approaches enable BitNet to converge effectively with larger learning rates compared to traditional FP16 Transformers, leading to improved training efficiency and stability, which is particularly beneficial in resource-constrained environments.   Comparative evaluations indicate that BitNet consistently outperforms existing quantization methods across a range of benchmarks, including zero-shot and few-shot tasks, while also maintaining lower inference costs. These findings highlight the advantages of quantization-aware training over post-training quantization, positioning BitNet as a prominent approach for developing efficient LLMs.    Arxiv: https://lnkd.in/e-D959-U

3

Last week’s DAC showcased exciting announcements from startups that are revolutionizing semiconductor design however these rising stars overshadowed a major update from an industry stalwart - Alphawave Semi - whose news is poised to accelerate the proliferation of chiplets and give rise to a “chiplet marketplace”. Alphawave Semi announced a breakthrough with the tapeout of the industry's first off-the-shelf, multi-protocol I/O connectivity chiplet. This technology can seamlessly integrate into larger SoCs or multi-chip modules, giving designers increased flexibility and more connectivity options for different applications. Reflecting on the Chiplet Summit this past spring, I remember listening to Tony Chan Carusone, Alphawave’s CTO, as he passionately described the significance of chiplets and how Alphawave will foster a more open and democratic design ecosystem. Now, just one quarter later, his team has launched a technology that brings this vision much closer to becoming a reality. Adding to their momentum, Alphawave also announced an expanded partnership with Samsung Foundry to advance the industry's progression toward cutting-edge semiconductor nodes, crucial for the evolution of advanced compute. Samsung’s roadmap in sub-4nm technologies includes FinFET and Gate-All-Around, which are designed to maximize performance in the post-Moore era. Initially targeted for data centers and HPC, these innovations may trickle down to low-cost, AI-enabled edge devices, putting the transformative power of AI into the hands of billions of people worldwide. This news follows Alphawave’s early June announcement of their new partnership with Arm, aimed at unlocking the potential of processors. Alphawave will give Arm access to their robust portfolio of connectivity solutions and silicon IP, which Arm will integrate into their proven portfolio of compute cores. At a time when AI technology is accelerating at breakneck speed, many companies are searching for ways to boost compute performance and address the challenge of moving data between compute and memory more efficiently. While multiple startups offer unique solutions, many customers want battle-tested technology to avoid the risk of investing in solutions that may become obsolete by the time they reach tapeout. Alphawave has developed vital solutions and in doing so, has increased design flexibility, built bridges to new process technologies, and shortened the path to market. “Connectors” are often the catalysts of innovation and Alphwave’s is a testament to that word. I’m encouraged by what I’ve seen from Alphawave and strongly believe they stand at the forefront of an emerging chiplet marketplace. More broadly, I believe their technology serves to democratize design technology and will give emerging companies more options to bring their brilliant ideas to life. #semiconductorindustry #ai #chiplets

112

Catching up with friends and investors in SF = solving the traveling salesman (founder) problem [1]. Does anyone have a quantum computer or slime mold colony [2] I can borrow to find a hotel that optimizes travel time to Palo Alto, SF, and Berkeley weighted by timing and # of meetings? 1. https://lnkd.in/eetRz2Va 2. https://lnkd.in/e2TQy8cH

12

5 Comments

Irreducible open-sources GPU accelerator for Binary Tower sumcheck. We wrote this article to describe how we tackle the main perf bottleneck for Binius prover.

24

2 Comments

Check out my conversation with Chris Saad on The Startup Podcast, talking about Positron AI's *now shipping* hardware with greater performance per dollar and performance per watt versus NVIDIA. https://lnkd.in/ghVUHgXm

67

3 Comments

You may find this interesting if you’re into #formalverification!

4

Another great article by Timothy Morgan at #NextPlatform "Because of the memory-compute imbalance, customers are spending a fortune on a device that cannot come close to its peak performance because of bandwidth and capacity bottlenecks in the HBM memory. We either need HBM4E memory early, or as we wrote about back in March this year when covering the simultaneous bi-directional NuLink PHYs from #Eliyan, we need a way to hook a lot more HBM3E memory to the current devices."

36

1 Comment

Tower Semiconductor Rides AI Surge with Silicon Photonics and Silicon Germanium Tower Semiconductor is leveraging the AI boom, with plans to double its silicon photonics revenue to $100M this year and potentially double it again in 2024, according to President Marco Racanelli. The Israeli chip foundry, ranked 7th globally, is using advanced silicon photonics and silicon germanium technologies to enable faster, more energy-efficient data transfers. The company boasts over 50 silicon photonics customers, including seven of the top 11 datacom transceiver makers, and holds a market-leading >50% share in transimpedance amplifiers (TIAs) and drivers. Analysts suggest these technologies are crucial for AI-driven optical interconnects, especially as data centers transition to co-packaged optics. To expand capacity, Tower has partnered with Intel Corporation and STMicroelectronics, securing cost-effective fab access for its power management ICs and other products. This strategic move aligns with increasing Western demand amid geopolitical shifts away from Taiwan and China. With silicon photonics projected to account for 10% of Tower’s revenue within two years, the company is well-positioned in the AI-driven semiconductor market. What are your thoughts on Tower’s growth strategy in the photonics space? Thanks again to EE Times | Electronic Engineering Times for the full article with more background and insights click the source link in the comments below. #Semiconductors #SiliconPhotonics #AIChips #DataCenters #ChipIndustry #OpticalInterconnects #TowerSemiconductor #TechInnovation

155

1 Comment