Gaofeng Huang

This week at IEEE #ISMAR2024 we present two foundational workpieces on Avatar Collaboration: #AvatarPilot and #MRTransformer. Combined they solve the issue of N people collaborating in dissimilar spaces. Specially non-trivial as when we add more people to a VR space the chance of their real spaces overlapping tends to Zero. The engine: #AvatarPilot Enables a person to point, grab or walk towards an object and remap that action to do the exact same pointing/grabbing/walking in a remote location, even if the objects are differently arranged. Super smooth. The secret sauce for this wonderful bijective algebra that allows N people to collaborate and move in remote locations is fully described on the paper “AvatarPilot: Decoupling one-to-one motions from their semantics with weighted interpolations.” Paper: https://lnkd.in/d3DAQRqd Video: https://lnkd.in/dBDMfWaV The deep evaluation: #MRTransformer Was then carried at Cornell University by Prof. Andrea Stevenson Won (Cheng-Yao (Eric) Wang and team), who fully implemented our mathematical approach inside a real AR setting “MRTransformer: Transforming Avatar Non-verbal Behavior for Remote MR Collaboration in Incongruent Spaces” Paper: https://lnkd.in/dD_hmcDc Video: https://lnkd.in/dXWxbc4n Authors: Cheng-Yao (Eric) Wang, Eyal Ofek, Hyunju Kim, Payod Panda, Andrea Stevenson Won, Mar Gonzalez-Franco

76

Our NVIDIA TAO (Train-Adapt-Optimize) team has developed scores of pre-trained models for computer vision or multi-modal uses, and the TAO Toolkit itself can be used to finetune, transfer-learn, or domain-adapt these models to your specific use cases. And now you can apply for academic grant to get free use of substantial amount of GPUs in Cloud Services. So, professors... go where no man has gone before!

34

2 Comments

With NVIDIA NIM and NIM Agent Blueprints, medical researchers across the public sector can jump-start their adoption of state-of-the-art, optimized AI models to accelerate their work. 🔗 https://nvda.ws/3NnkMYy Learn how #NVIDIAInception startups Abridge & ConcertAI leverage NVIDIA AI solutions to improve efficiency, reduce administrative burdens, and advance medical research and patient care. 🩺 Abridge ConcertAI

2

Good overview of AI chip landscape with risks and challenges. https://lnkd.in/gndBHNcx A few thoughts that popped up in my head: 1️⃣ Scalable #AI Hardware: As language models grow larger and adopt architectures like Mixture of Experts (MoE), AI hardware must embrace similar principles, optimizing for modularity and efficiency. 2️⃣ Chiplet Revolution: Industry trends point to chiplets—smaller, specialized chips (e.g., I/O, memory, processing)—connected via high-bandwidth interconnects. This enables tailored solutions for bandwidth and latency needs. 3️⃣ Advances in Packaging: Once overlooked, packaging technologies like multi-die integration and package-on-package designs are now driving breakthroughs in AI performance. https://lnkd.in/g95T5dbT Lets take an example of an AI stack specializing in inference. #GPUs are inherently optimized for dense dot product operations, which do not perform efficiently when deploying a sparse model. This is where management of model weights in hardware becomes paramount. One of the broad approaches is to distill the #LLM to identify the neurons that get activated the most during inference, and load it in-memory caches. This reduces data transfers from flash storage to DRAM components. #Apple can be credited with this breakthrough: https://lnkd.in/gYRDaFem Another approach is looking at customizing GPU workloads either through kernel tuning or clever compiling. Some of the cool work I have read to overcome: 1. Load balancing:  GPUs are made up of an array of streaming multiprocessors (SMs). Each SM is in turn made of a wave of thread blocks that run concurrently. Each SM has a dedicated L1 cache and a shared memory space that is L2 cache. In a sparse matrix, a number of techniques have been tried to manage imbalance between thread blocks and within it. 2. Megablocks: Transform a sparse MoE as a single block-sparse matrix multiplication instead of multiple dense matrix multiplications by leveraging a blocked compressed sparse row (BCSR) format. The resulting algorithm handles the dynamic computational demands in sparse MoE without the need for load balancing, padding, or token dropping.

11

3 Comments

Hallucinations in LLMs are inevitable, no seriously! Josh Fritzsche and I have worked hard at Voze to find a way to tune, prompt and even plead 🙏 with LLMs to prevent them from the occasional 🍄 hallucination. It seemed like the better we have done the harder the edge cases have been to solve and eventually we realized that they tend to frequently happen when LLMs predict or make anticipations on "nothing". I remember at one point nearly going insane with frustration because when LLMs had a hallucination, they were EPIC! 😱 Then we pushed through it and benefited for the effort! How did we solve this common problem? We didn't (Not completely), we got better at identifying the issues as they happened. Once we identified in the data we were able to start looking for patterns. The same catalysts that cause it to happen once if left unchanged caused it over and over again. Our solution is astoundingly simple. Once you know that it's likely to happen..... let it! Accept it will 🍄, but also be preemptive in not focusing on it and use alternative data sources, generators or providers. Change the conditions that cause it but continue to monitor it as it happens. As Dan Caffee is fond of saying to me on a regular bases. "Dusty we need the system to be adaptive and provide mechanisms to learn from it's past." Well he is right! All we needed was data and the people that contribute to it's quality. Huge props 👏 to strong support from Kathy and Janelles audit teams. We were armed with a knowledge and a desire to learn from past mistakes. Armed with confidence that we could correct these problems we could focus on the data to prevent their impact going forward. When doing millions of LLM (Generative AI) calls regularly Josh and I have learned that hallucinations will happen. We also know exactly how we will handle and prevent it going forward! Failure is not doing anything and accepting defeat.

4

2 Comments

🎇 On National Day, I went for a leisurely drive in San Francisco and ended up "stress-testing" a Waymo self-driving car on the road. 🚗 As an autonomous driving practitioner, who wouldn't be curious about the real-time robustness of the cutting-edge Waymo One driving system? While cruising downtown, I accidentally noticed a Waymo car tailgating me. While this isn't unusual for SF citizens, a wild, "evil" idea suddenly hit me: Why not directly "adversarially attack" the world's autonomous driving status quo? I executed an unexpected maneuver—suddenly reversing—to see how it would react. 🌟 The response was stellar! At the moment I reversed, Waymo One honked instantly—quicker than any human could—activated its hazard lights, and backed away to maintain a safe distance. This reckless move on my part served as an edge case to test the algorithm's robustness under extreme conditions and, potentially, could be a challenging training sample to enhance Waymo's future autonomous systems. 💫 Thrilled to personally experience how current cutting-edge autonomous algorithms handle rare driving behaviors—and how stable and safe Level 4 autonomy is in dealing with diverse scenarios. However, it also prompted deep reflection as an AI researcher in this field: 🤔 In an industry with little room for error, how can we ultimately avoid or minimize issues that AI fails to handle? 💡 I believe two research directions are particularly promising for achieving Level 5 level autonomy in future mobility systems: - 1️⃣ Development and deployment of vehicle-to-everything (V2X) cooperative systems (including V2V, V2I, V2P, etc). Our initial studies (e.g., V2X-ViT, ECCV'2022 arxiv.org/abs/2203.10638) show that in scenarios with severe occlusions or noise, such cooperative systems can significantly enhance the robustness of perception systems, thereby eventually improving traffic safety. - 2️⃣ Adversarial scenarios generation (including Sim-to-Real, generative modeling). Research done by my colleagues at UCLA (V2XP-ASG, ICRA'2023 https://lnkd.in/gu5nKVHD) shows that adversarial learning techniques can effectively simulate adversarial scenarios, greatly improving model robustness in complex "corner case" situations. Of course, it's often infeasible to collect such collision scenarios. 👨‍🏫 As a new Assistant Professor of CS at Texas A&M University, I will lead a group focusing on these exciting research directions, which can be a proactive approach to reducing accidents and improving safety for all. 🔥 I look forward to future collaborations with governments, academics, and companies to research and develop data and algorithms that can help enhance the safety of vulnerable road users, especially seniors and children. We envision a people-centered intelligent transportation system in the future. Interested in these topics? Let's connect and discuss further! #AutonomousVehicles #AI #MachineLearning #SmartCities #Transporation #Humanity #Mobility

144

3 Comments

Thats the exact explanation I gave to a group of investors last week. (… it’s just Precision/Recall, attributes vectorization as “embedding”, K-Nearest Neighbor, User Search Intent, DAG, ML algorithms, etc all over again …). LOL and they asked what’s new this time around. And I was like not much, ML is still the backbone, and AI is the inference and actions based on the model. It works better when there is a feedback loop in a closed system. But the current state of the art like ChatGPT is based on scraping the Internet (garbage in, garbage out). Kinda like early search engines. What is a better analogy is that NVIDIA/Snowflake etc are selling tools, aka pickaxes to prospectors. The gold is up to you to find. And the opportunity right now is to monetize industry specific (healthcare, travel, financial services) clean data to build models with (aka a treasure map for the prospectors). From an enterprise perspective, what’s out there today aka generative AI isn’t very useful. Too much hype. To build useful engines will require an effort similar to search revolution 2 decades ago. From

16

3 Comments

I am excited to share the latest blog, co-authored by myself, Junhwi Kim and Vyas Sekar, detailing how we managed to successfully localized and improved LLM by combining Fine-Tuning and RAG. https://lnkd.in/gKDkgGPq

70

3 Comments

Come see this talk, which I view as the culmination of several years of discussions and co-teaching with George Fairbanks. It's a bit of an overview, a bit of an explainer, a bit of a how-to, but mostly a sneaky way to make some important vocabulary from design become load-bearing in how you think about software testing.

86

1 Comment

Insightful post by Pat Walters, if you had any doubt that chemistry and biology need specialized models, I hope this clears it. these models need specialized tokens and goals, at Converge Bio we are successfully using specialized models to augment biological R&D

10

1 Comment

We are thrilled to highlight the last of our team of interns; Arvind Swamynathan, our Machine Learning Intern at Arrow Diagnostics this summer! Arvind is an incoming senior at UC Berkeley, where he's majoring in Bioengineering and minoring in EECS (Electrical Engineering/Computer Science) and Data Science. His passion for applying software and hardware solutions to improve health and quality of life, particularly in medical device/biosensor and medical imaging engineering, aligns perfectly with our mission at Arrow Dx. #TeamArrowDx #MachineLearning #BiotechCareers"

37

It is time to look forward to the summer of 2025! https://smrtr.io/npvQ- Staff Machine Learning Scientist, Intern - Summer 2025 at Visa, inc.   Unleash the world scientist within you this summer! Inspired by Richard P. Feynman's maxim, “You do any problem that you can, regardless of field,” we're inviting you to contribute your innovative skills in cross-industry advancement of science and humanity at VISA, the technology giant at the forefront of billions of global credit payments. We’re calling all physical scientists to venture into our world of ML/AI. The mission? To uncover fundamental truths hidden in the vast and largely untapped domain of Payment transactions, a realm as complex and intriguing as Physical interactions. Just as scientists construct effective models to make sense of real-world data, we strive to unearth elemental factors driving billions of transactions per day. With the rise of neural network-based deep learning in Physics, we're pioneering the use of these techniques to revolutionize global commerce.

121

1 Comment

Join us in Irvine on May 15th at 6:30 PM for our latest OC ACM talk, "Understanding Life via Computational Bioinformatics" presented by Dr. Shawn X. Wang, a distinguished professor of computer science at Cal State Fullerton. For more info or to RSVP click here: https://lnkd.in/gX72Ss_g Dr. Shawn X. Wang will provide an insightful exploration of computational bioinformatics, a field dedicated to extracting, organizing, and interpreting biological data to uncover new insights. Divided into two parts, the presentation will first highlight the transformative impact of computational bioinformatics research on enhancing quality of life and deepening our understanding of biology. In the second segment, Dr. Wang will offer a concise overview of his research endeavors, focusing on innovative algorithms and data analysis techniques applied to protein 3D structures, microarrays, and DNA sequence comparisons. Join us for a cohesive journey into the intersection of computation and biology. #acm #compsci #bioinformatics #techtalk #irvine #calstatefullerton #science #innovation

1

I appreciate the insight from Prof. Shashua to highlight the pos and cons of E2E and abstraction. I think that Tesla took the route of Data Center + Data Engine + E2E on the edge while minimizing engineers' intervention. It is not necessarily the best route for the other competitors. In other market, engineers' intervention in the design could be greater or better than the data engine at that time.

2

As AI models move from research to production, they will impact the lives of billions of people worldwide. It is critical to devise methods that can help ensure that these models are fair and do not exhibit harmful social biases. Our Multimodal Cognitive AI team at Intel Labs has developed methods to probe and mitigate intersectional social biases in multimodal foundation models. In our CVPR'24 paper, we report how large-scale AI clusters powered by Intel Gaudi2 in Intel Developer Cloud (cloud.intel.com) are used to create synthetic counterfactual image-text pairs that allow probing social biases in multimodal foundation models. Furthermore, we show how we can mitigate these intersectional social biases by fine-tuning foundation models on our synthetic data. To further support research in social biases in multimodal foundation models, we fully open-source our synthetic dataset created on Intel Dev Cloud AI supercomputing cluster. 👉 SocialCounterfactual Project Page: Read our CVPR'24 paper and access our synthetic dataset: https://lnkd.in/gMZKwazZ 👉 Intel Developer Cloud for large-scale AI clusters powered by Intel Gaudi: cloud.intel.com Looking forward to how our Multimodal Cognitive AI team continues to push the state-of-the-art in AI, while continuing to support open-source datasets and code and striving towards how we develop AI systems in a responsible and ethical manner. #intelai

88

1 Comment

🔥 Create Waves, Not Ripples: Define What’s Next at NVIDIA, Waymo, and Microsoft! 👋 Hello Game-Changers! Why be ordinary when you can go viral? The future is all about creating impact that spreads, and this week’s roundup has everything you need: AI/ML roles, tools to boost your influence, and career hacks to make your next move unforgettable. 🤖 AI/ML: Lead the Revolution Power the future with intelligence. These roles are tailor-made for those who think bigger, bolder, and smarter. 1️⃣ Machine Learning Engineer🏢 Wonder | 📍 New York, NY | 💰 Competitive 2️⃣ 2025 Summer Intern – Perception, Few-Shot Learning🏢 Waymo | 📍 Mountain View, CA | 💰 $100k/year 3️⃣ Research Scientist – Programming Systems (New Grad)🏢 NVIDIA | 📍 Santa Clara, CA | 💰 $160k–$253k/year 4️⃣ AI/ML Product Analyst🏢 Leidos | 📍 Remote | 💰 $87.1k–$157.45k/year 5️⃣ Machine Learning Engineer Intern – Hardware Optimization🏢 Hugging Face | 📍 Remote | 💰 Competitive Unlock the Power of Your Toolkit Streamline your workflow with these must-have tools: Google Colab: Collaborate on Python in the cloud. TabNine: AI-powered code completion. DVC: Version control for ML projects. 💡 Pro Tip: Spend 30 minutes exploring a feature or building a mini-project today. #CreateImpact #DefineWhatsNext #AIRevolution #DriveInnovation #BoostYourToolkit #CareerGoals #FutureOfBusiness #LeadershipJourney #TechGameChangers #NVIDIA #Waymo #Microsoft

1