Wenjun Gao

sharpening writing skill is one of killer skill!

2

Massive news for AI enthusiasts (and anyone looking to give their business a serious edge)! Did you hear about Google's new Tensor Processing Unit (TPU) v6, codenamed Trillium? This is a game-changer for AI processing power. Ready to supercharge your AI efforts? ⚡️ Reach out to the team CloudWerx today, and let's chat about how TPUs can revolutionize your business!

20

Using generative AI, NVIDIA operations built an AI Planner agent, developed on NVIDIA Inference Microservices (NIM). The agent leverages LLM, NeMo Retriever and CuOpt NIM to reduce re-planning time from hours to just seconds. https://nvda.ws/4bIfS2T

2

7 ways to slash latency in System Design: When I worked on Meta's distributed data store, managing massive data wasn't the most challenging part – fighting latency was. After years of trial and error, I'm sharing 7 proven best practices for building low-latency systems. 1. Choose the correct architecture Swap slower monolithic architectures for modular, scalable systems that cut latency by operating independently. 2. Optimize data management Choose SQL for structured data or NoSQL for flexibility, and use indexing, sharding, and replication to speed up data retrieval. 3. Improve network design Reduce data travel with CDNs, load balancers, and edge caching. 4. Use efficient communication protocols Use fast protocols like HTTP/2, WebSockets, and gRPC to streamline communication and cut round-trip times. 5. Optimize your code Use efficient algorithms, data structures, parallel processing, and I/O reduction to speed up execution and reduce bottlenecks. 6. Use high-performance hardware Opt for SSDs over HDDs and low-latency cloud services like AWS Global Accelerator for faster performance. 7. Implement smart caching strategies Cache at multiple layers with LRU/LFU eviction and sync updates to avoid stale data. Read the full blog for a closer look at these 7 practices and how to implement them. (Your system—and users—will thank you). https://educat.tv/3B5Dh0N #SystemDesign #SoftwareEngineer

20

Over the past few years, I’ve consistently advocated for predictable large-scale training, leveraging scaling laws and Maximal Update Parametrization (muP). This approach is backed by compelling evidence from our brilliant ML team at Cerebras. Before every large-scale run, we always fit a scaling law to guide our decisions. For optimal hyperparameters and training stability, we rely on muP and mu-transfer, allowing wide hyperparameter search at small scale and successful hyperparameter transfer to larger model sizes. We've been using muP on Cerebras hardware with our ModelZoo codebase (https://lnkd.in/gvnQzAvk) all this time. Recently, Nolan Dey and Joel Hestness teamed up with Quentin Anthony from EleutherAI to create a practical guide on muP for the community, alongside a port of muP to the popular nanoGPT library (https://lnkd.in/gYnpbmD4). Check it out at https://lnkd.in/gijgxmXp !

109

🎯 How teams are cutting Machine Learning Engineer hiring time from 90+ days to 28 days: 1. Role-specific technical assessments that mirror real projects • Real-world problem-solving instead of whiteboard exercises • Actual code reviews from your codebase • System design challenges based on your architecture • Practical ML model optimization tasks 2. Direct access to tech leads in first interview • No HR screening delays • Technical discussion from day one • Immediate architecture alignment • Quick technical fit assessment 3. 48-hour feedback commitment from all parties • Same-day interview feedback • Next-steps within 24 hours • Offer decisions in 48 hours • Clear communication timeline 4. Streamlined technical evaluation process • One comprehensive technical round • Parallel evaluations when possible • Combined culture/technical assessments • Quick technical team consensus Teams implementing even one of these cut hiring time by 40%. "Especially effective for fast-growing startups and scale-ups" #TechHiring #MLEngineering #TechRecruitment

3

“Google was built for this moment. We’ve been pioneering GPUs for more than a decade,” Google CEO Sundar Pichai Introducing Trillium, Google's sixth-generation Tensor Processing Unit (TPU). Trillium was specifically designed to accelerate the development and deployment of advanced AI models. With significant improvements in compute power, memory, and networking capabilities Key advancements in Trillium include: ➡ Enhanced compute performance: 4.7X increase per chip, enabling faster model training and serving. ➡Increased memory and bandwidth: Doubled High Bandwidth Memory (HBM) capacity and bandwidth, along with Interchip Interconnect (ICI) bandwidth, for handling larger models and datasets. ➡SparseCore accelerator: third-generation SparseCore, Specialized for processing ultra-large embeddings common in advanced AI workloads. ➡Sustainability: 67% more energy-efficient than the previous generation. 👉 👉👉👉 https://lnkd.in/dyASZGWy #AI #ML

2

1 Comment

Just did a keynote in PyTorch conference, on LLM deployment. Thanks for the great work of PyTorch team, our partners, and the interests from the audience! Please try out our deployment ecosystem of torchchat, torch.compile, torch.export, AOT Inductor, torchao and ExecuTorch, and checkout all the amazing talks, posters and demos in the conference!

50

3 Comments

We’ve run almost a billion inference requests at Beam Here are a few things we’ve learned about real-time inference APIs 1. For small models, you can often run multiple workers in parallel on the same GPU and increase throughput 2. Weights should be cached on disk or in distributed storage. You don't want to be downloading large files from cloud storage during an inference request 3. Cross-region latency matters. Regional caching is hard and requires a lot of planning 4. You can't use the same autoscaling strategy for every model. You have to see how your workload performs in the real world and choose a strategy that makes sense When you run APIs powered by large ML models, performance isn’t guaranteed. What other strategies have you employed in your ML pipelines to optimize performance?

37

1 Comment

𝗡𝗩𝗜𝗗𝗜𝗔 𝗔𝗺𝗽𝗲𝗿𝗲, 𝗛𝗼𝗽𝗽𝗲𝗿, 𝗮𝗻𝗱 𝗕𝗹𝗮𝗰𝗸𝘄𝗲𝗹𝗹 𝗚𝗣𝗨𝘀: 𝗪𝗵𝗮𝘁'𝘀 𝗡𝗲𝘄 𝗳𝗼𝗿 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴? NVIDIA's latest GPUs bring powerful upgrades for ML workloads: • 𝗔𝗺𝗽𝗲𝗿𝗲 𝗔𝟭𝟬𝟬: Introduces 𝗧𝗙𝟯𝟮 and 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱 𝘀𝗽𝗮𝗿𝘀𝗶𝘁𝘆, accelerating deep learning efficiency. • 𝗛𝗼𝗽𝗽𝗲𝗿 𝗛𝟭𝟬𝟬: Adds 𝗙𝗣𝟴 𝗽𝗿𝗲𝗰𝗶𝘀𝗶𝗼𝗻 and a 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗲𝗿 𝗘𝗻𝗴𝗶𝗻𝗲, boosting AI training by up to 9x with faster data transfers. • 𝗕𝗹𝗮𝗰𝗸𝘄𝗲𝗹𝗹 𝗚𝗕𝟮𝟬𝟬: Features 𝗙𝗣𝟰 𝗽𝗿𝗲𝗰𝗶𝘀𝗶𝗼𝗻 and a hybrid 𝗚𝗣𝗨+𝗖𝗣𝗨 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲, improving speed and reducing latency for large AI models. Each generation pushes the envelope for precision, performance, and scalability in AI. Read my full article for a deeper dive: https://lnkd.in/gmRDpKsc #NVIDIA #GPUs #DeepLearning #MachineLearning

14

Amazon continues to push the boundaries of large language models and conversational AI. At this year's ACL conference, Amazon researchers presented over 30 papers exploring innovative techniques to enhance language models - from improving model evaluation and robustness, to leveraging knowledge graphs for better reasoning, to advancing applications like code synthesis and multimodal understanding. This breadth of cutting-edge work showcases Amazon's deep commitment to advancing the state-of-the-art in natural language processing and building the conversational AI systems of the future. The company's world-class AI research continues to have a major impact on both the academic and industry landscapes. Looking to join a team at the forefront of the conversational AI revolution? Check out the latest openings on the Amazon Science careers page. #AmazonScience #ConversationalAI #NaturalLanguageProcessing #MachineLearning #ACL2024

3

Insightful Paper on Scaling law for Precision TL;DR The more tokens you train on, the more precision you need for inference. The paper studies the limits of quantization that has implications on compute resources, model size decisions. - Training in lower precision reduces a model's effective parameter count. - Post-training quantization degrades performance more for models trained on larger datasets. - Paper also suggested a scaling laws to predict the loss of a model with different parts in different precisions, and suggest that training larger models in lower precision may be compute optimal. https://lnkd.in/gk8yZx_p #quantization #LLMs #AI #inference

13

On my learning some features make GPUs the backbone of modern ML, what you say ? --SIMT Architecture GPUs use the Single Instruction Multiple Threads (SIMT) model, grouping threads into warps (e.g., 32 threads in NVIDIA GPUs). Each warp executes instructions in lockstep, allowing simultaneous processing of vectorized operations like matrix multiplication. --High Core Density: A typical GPU (e.g., NVIDIA A100) has thousands of CUDA cores distributed across Streaming Multiprocessors (SMs), enabling concurrent execution of many small, independent tasks in neural network computations. --Tensor Cores for Mixed Precision: Modern GPUs feature Tensor Cores, which specialize in fused multiply-add (FMA) operations, essential for accelerating FP16/FP32 matrix multiplications. Tensor Cores are optimized for AI tasks like training transformers and CNNs. --Memory Bandwidth Optimization: GPUs leverage HBM (High-Bandwidth Memory) or GDDR6 for fast data transfer between global memory and cores. NVIDIA A100 delivers up to 1.6 TB/s, enabling efficient handling of large tensor operations. --Shared and Cache Memory: GPUs include a shared memory pool per SM and a multi-level cache hierarchy. Shared memory is low-latency and ideal for thread communication within a block, critical for synchronizing kernel operations. --Occupancy and Thread Divergence: High GPU occupancy ensures that idle threads due to memory stalls or warp divergence (branching) are masked by scheduling other active warps. This dynamic scheduling boosts throughput for complex models. --Sparse Matrix Optimization: Advanced GPUs (e.g., NVIDIA H100) include hardware support for sparse matrices, which reduces computation and memory requirements by leveraging sparsity patterns in ML models. --CUDA Execution Model: CUDA's hierarchical thread-block-grid structure maps computational tasks to the GPU hardware. Developers can explicitly control memory usage, register allocation, and kernel execution to optimize performance. --Multi-GPU and NVLink: Multi-GPU systems connected via NVLink provide high-bandwidth, low-latency communication, enabling large-scale distributed training . FP64 and BFLOAT16 Support: Specialized GPUs support FP64 for scientific computations and BFLOAT16 for AI workloads, balancing precision and performance. BFLOAT16 reduces memory usage and speeds up training without significant accuracy loss. #keepMatrixing

1

𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿 𝗘𝘃𝗮𝗻𝗴𝗲𝗹𝗶𝘀𝗺 is a unique function that fills a lot of niches in developer relations & product marketing. But one particularly helpful intersection is around technical product launches. Here at Intel, when it’s time to launch new silicon for AI inferencing, it typically goes hand in hand with a new version of OpenVINO, our open source AI inferencing acceleration software to enable that silicon. So we need launch blogs, samples, and presentation material ready for developers when the new hardware and software launch. Evangelists are the ones on my team who write this content, complementing launch momentum/awareness by giving developers a CTA they can do right away on the new platform. Evangelists on my team typically will spend months playing with the new hardware in pre-release form, along with pre-release software, to build their own learning and prepare the launch content. The interesting thing is that making them “part of the development/validation team” in this way, they often find many bugs (and workarounds) in software, drivers, and libraries. Many of these get fixed before launch; of course some don’t. This is both exciting for them to “get new hardware early” and frankly quite frustrating to them at times as the kinks are worked out! Their work then becomes a key location of learnings to (a) help answer developer questions, (b) demo the new platform to best effect, (c) write launch content. In this way, Evangelism intersects marketing, engineering, customer support, and validation. Have evangelists on your team? Put them in the middle of your next product launch, and give them access to the product two months ahead of time! #openvino #developerrelations #productmarketing #iamintel

32

2 Comments