Long Gong

It’s been a pleasure to talk alongside such great researchers from Google DeepMind and top Taiwanese universities at Google AI Taiwan Academic Forum. (Details at https://lnkd.in/gnbcWZyf) My topic — “Challenges of Applied ML” — covered 4 observations: - We ought to build product, not merely expose LLMs. - Making them useful and trustworthy requires handling input and output properly. (Read more at https://lnkd.in/gcNDC54R) - Making those model useful requires high-quality data for tuning (and evaluation). (More at https://lnkd.in/gyVYQaaD) - There’s many practical deployment considerations, one of the main ones being cost.

110

Transforming into a Generative AI career (business) #2 We typically begin with enterprise use cases, adding value by modernizing business domains as we build our Generative AI skills portfolio. This includes areas such as Knowledge Intelligence, Conversational Services, Copilots and Product Design etc., which lead to the development of Generative AI technology stacks. #Nvidia offers a more disruptive vision for the future of the enterprise through the concept of the AI agentic digital employee (robot). #Accenture has invested $3 billion to build a managed services business on top of Nvidia’s technology stack (see the recent interview with Jensen Huang and Julie Sweet). Nvidia's AI Factory needs three computers (OSs) for the Gen AI project: * Nvidia AI Supercomputer: For training the Model * Nvidia Jetson Orin, CUDA, NIM, Nemo Stack: For running the model * Nvidia Omniverse Platform: For learning and refining the skills from the simulated scenes What have been your thoughts on positioning Nvidia technology in your Generative AI skills portfolio? ______ I’m sharing how I’m navigating from a Solutions Architect into a Generative AI career. Share your perspectives, and let’s make this journey even more exciting together! #artificialintelligence #generativeai #innovation #LLM #Transformer #RAG

8

4 Comments

Researchers from UC Berkeley, UIUC, and NYU have proposed a new framework to enhance the decision-making capabilities of Large Vision-Language Models (VLMs) using Reinforcement Learning. Key highlights include: - Reinforcement Learning Fine-Tuning: The team introduced a method to fine-tune VLMs with Reinforcement Learning, enabling models to learn from interactive environments and make optimal decisions in multi-step tasks. - Chain-of-Thought Reasoning: By incorporating Chain-of-Thought (CoT) reasoning, the VLMs efficiently explore intermediate reasoning steps, significantly boosting their performance in decision-making tasks. - Outperforming Commercial Models: This framework appears to allow a 7B VLM to surpass well-known commercial models like GPT-4V and Gemini, showcasing substantial improvements in both arithmetic and visual semantic reasoning tasks. You can find the paper here: https://lnkd.in/gs-tBqKk This is one of many new innovative techniques for maximizing performance in smaller models. I'm sure we'll be many more in the coming months! #GenAI #AI #ReinforcementLearning #VLM #MachineLearning HOPPR

16

1 Comment

Trying to reduce costs of your LLMs by 2x? Use a small highly optimized "router" LLM to decide if you need a heavy or a light LLM based on the context. Simple questions go to the light LLM, tougher ones to the heavy LLM. Most surprising finding: the router LLM generalizes across different heavy and light LLMs -- no fine tuning required! (Also this is my first academic paper in 20 years -- it was a lot of fun working with Amjad Almahairi on this)

113

6 Comments

Collaborative AI is gaining momentum at both MNC and Startups, they are investing into this EvolutionNext Robotics AI Transformative Experiential Learning Journey. I am extremely glad to put Singapore on the World map, by anchoring Robotics AI companies here, creating Jobs of the FutureNow as Roboticist. Do support my Leadership Initiative for Singapore as National Interest; help me enable our next-generation Singaporeans to take on Science, Technology, Engineering and Mathematics STEM Education and longer-term Career Practice into Industry 5.0 as Machine Intelligence of Human-Robot Collaborative, Human-centric Autonomy. * My STEM OneCareer Pinnacle and Finale as a 55-years old, always just another very ordinary Singapore-born Singaporean Scientist Engineer in Singapore, essentially a Nobody here 🌸

13

2 Comments

During my short time with NVIDIA's Display Drivers team, I had the opportunity to build a chatbot that helped developers quickly access information related to sanity infrastructure. I also attended demos, like NVIDIA's ChatRTX(https://lnkd.in/gsU4x7Vt), that were building similar RAG chatbots. Recently, I found a course, that covered the challenges I faced such as pre-processing diverse data sources and retaining memory context, in just an hour(or a weekend if you code along). Things are moving quite fast for LLMs. While looking for resources to further bridge my knowledge gaps and work on another project, I came across another gem from DeepLearning.AI. For those interested in learning about LLMs, I highly recommend(in the same order): 1. Generative AI with LLMs(https://lnkd.in/guQz4Mcc): Key LLM concepts and hands-on experience via a Dialogue Summarization project. 2. LangChain Chat with Your Data(https://lnkd.in/gRUQVdSR): A practical guide to building chatbots that interact with your data, covering common challenges and solutions. #llm #machinelearning #deeplearning #AI #artificialintelligence #generativeai

36

Excited to share the work that Chris Glaze Changho Shin Frederic Sala and I did on long context LLM evaluations. While there’s plenty of problems with LLM evals, a recurring one we’ve seen at Snorkel, working with enterprise clients, is having to rely on far too simple/toy benchmarks that do not relate to business problems. It is why we developed SWiM, a framework that lets users test long context model capabilities directly on their own data and tasks. Read the paper here: https://lnkd.in/g3uSuKcW

31

Just did a keynote in PyTorch conference, on LLM deployment. Thanks for the great work of PyTorch team, our partners, and the interests from the audience! Please try out our deployment ecosystem of torchchat, torch.compile, torch.export, AOT Inductor, torchao and ExecuTorch, and checkout all the amazing talks, posters and demos in the conference!

50

3 Comments

Back in May, I was invited by the local chapter of IEEE SSCS in Santa Clara to give a talk on how to use LLM and Gen AI for high-speed design. The video has been uploaded to YouTube. It is a good summary of how we are using LLM and AI for SI. https://lnkd.in/eUy2HDhB A few months later, Eric Schmidt gave a controversial talk on where he thought AI would progress in the next few years in a Stanford computer science class. He was a little too frank on telling students to steal TikTok IPs and why he thought Google would fail. https://lnkd.in/epMKkmvv Both talks seem to completely overlap on how to create domain knowledge dominance using advanced LLM. He gave examples on taking over TikTok and in chemical engineering, I was focusing on high-speed SI designs. But it can be applied to any other hardware design disciplines.  What we are pursuing, and he seems to suggest the same approach : 1) Use LLM with deep context window to scan through large volume of knowledge (text, video or sound), in my case for SI/PI we will be scanning large volumes of design specs for design domain knowledge 2) Use LLM agents to execute task based on the ingested knowledge (context to action), in my case, we will be initiating simulations based on design spec range and look for optimal solutions 3) Repeat the above with multiple chain of thoughts and store the observation to draw a conclusion, in my case, after scanning through the design specs and obtain optimal spacing for performance, generate routing and constraint rules 4) We also add an optimization step which is necessary for engineering  5) Improve the LLM agent performance using an adversarial agent to critique the LLM agent's performance 6) We also add using fine-tuning and transfer learning to future proof the domain specific models. I apologize both are long talks ~1 hour.  I hope this will be a good starting point to discuss the use of LLM and Gen AI for SI/PI. Feedback and comments are most welcome.

13

2 Comments

"Why did Santa's AI-driven sleigh keep running into trees?" So much 🔥 from Julie Wang and John Allard in today’s OpenAI live stream! They covered Reinforcement Fine-Tuning, medical research, and shared some truly punny jokes. Moments like these remind me why I love my team: brilliant people solving fascinating problems—and having fun along the way! https://lnkd.in/gJnNiTXc If you'd like to participate in the Research Program by sharing data, check out https://lnkd.in/gZYehx9T or ping me. Reinforcement Fine-Tuning is a new technique for teaching the new models how to reason. It's kinda like how AlphaGo taught a model how to play by "grading" games, then learning what "good" moves are by comparing moves in winning vs losing games. This opens up a whole new paradigm for models, where you're not just teaching them the answer, but teaching them how to arrive at the right answer. What is Reinforcement Fine-Tuning (RFT)? It’s a groundbreaking technique for teaching models how to reason. Think of it like AlphaGo: the model learns by being “graded” on games and understanding what makes moves in winning games better than losing ones. With RFT, prompts are like the “games” and chain-of-thought reasoning is like the “moves.” By evaluating the "winning" reasoning steps, the model learns which ones lead to better answers. This means we’re not just teaching models the answers—we’re teaching them how to think through problems and consistently arrive at the right solutions. My team’s mission is simple: give you the world’s best model for your domain. If you’re finding it hard to get API responses to work for your business—whether it’s a performance issue, cost concern, or accuracy challenge—let me know. We have tools to improve outcomes, and I love working with customers to see how we can do even more. Oh, and Santa’s sleigh? Julie gave us the answer: “Because he hadn’t ‘pine’-tuned yet!” 🥁

22

1 Comment

Super excited about this. I designed Venice CDC for #venicedb about two years ago and now it's powering AI on GPU's at LinkedIn! The team working on LiNR talk about it in the linked paper are presenting at #CIKM this week. Check it out! #ai #gpu

30

1 Comment

🚀 Exploring Prime Gaps with Cutting-Edge AI Techniques 🔢 Prime numbers have captivated mathematicians for centuries, but their mysteries remain as enticing as ever. Yitang Zhang's groundbreaking work on bounded prime gaps brought a seismic shift to number theory, proving the infinitude of primes within a finite gap. 🌌 Building on Zhang's remarkable achievement, I’ve taken a computational leap forward. Using Hyperdimensional Computing (HDC) and Neuro-Symbolic AI (NSAI), I’ve explored prime gaps in a way that blends mathematical elegance with the power of advanced AI. From encoding prime numbers as high-dimensional hypervectors to visualizing patterns in interactive 3D, this hybrid approach enhances both computational efficiency and interpretability. Highlights include: ➡️ Advanced sieve algorithms for prime generation ➡️ Machine learning models predicting prime gaps ➡️ Neuro-symbolic reasoning for explainable insights ➡️ Clustering and dimensionality reduction revealing hidden patterns If you’re curious about how modern AI can unravel age-old mathematical mysteries, check out my latest blog post. Together, we’ll dive into the fascinating intersection of primes and AI innovation. 💡 🔗 Read here #PrimeNumbers #HyperdimensionalComputing #NeuroSymbolicAI #MachineLearning #AIResearch #NumberTheory #Innovation #LinkedInLearning

1

1 Comment

It's hard to run inference for llms because of their large memory footprint because of all model parameters and activations required at inference time and parallelization is hard because of the autoregressive nature of generating tokens.    So to optimize inference in general you need to lower the memory footprint (use fewer GPUs), compute(FLOPS) and latency.      There are some approaches to make inference efficient(less memory usage and fast):      1. Apply parallelism techniques to run inference across a large number of GPUs      2. Offloading unused data to the CPU and reading it back when needed (less memory usage but high latency)      3. Smart batching strategies like continuous batching (vllm uses this strategy)      4. Compression techniques like quantization, distillation or pruning    5. Architectural optimizations like using PagedAttention (efficiently manages key and value in memory)

14

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts https://lnkd.in/gb78Y4bn In the latest foray into the world of multimodal language models, researchers have unveiled an intriguing new approach they've dubbed "CuMo." This novel technique aims to harness the power of Mixture-of-Experts (MoE) architectures, incorporating Co-upcycled Top-K sparsely-gated MoE blocks into the vision encoder and the multi-layer perceptron connector of multimodal LLMs. The incorporation of these MoE components is a bold move, promising to enhance the capabilities of multimodal LLMs in a significant way. However, the researchers didn't stop there. They've also adopted a three-stage training approach that employs auxiliary losses to help stabilize the training process and ensure a balanced distribution of workload across the expert modules. One of the most noteworthy aspects of this project is that CuMo has been exclusively trained on open-source datasets, a commendable decision that promotes transparency and accessibility in the field of AI research. Overall, this project presents an exciting exploration of MoE architectures in the context of multimodal language models. While the true impact of CuMo remains to be seen, the innovative approaches employed and the promising early results make this a development worth keeping an eye on in the rapidly evolving field of AI.

3

I'm delighted to share that our #AutonomousDriving team @Samsung SAIT have two papers accepted to NeurIPS 2024! Congrats to the authors, and thanks for the great support from our incredible team. #NeurIPS #NeurIPS2024 [1] Unveiling the Hidden: Online Vectorized HD Map Construction with Clip-Level Token Interaction and Propagation Nayeon Kim*, Hongje Seong*, Daehyun Ji, and Sujin Jang In this paper, we introduce a novel paradigm of clip-level vectorized HD map construction, MapUnveiler, which explicitly unveils the occluded map elements within a clip input by relating dense image representations with efficient clip tokens. Our extensive experiments demonstrate that MapUnveiler acheives state-of-the-art performance on both the nuScenes and Argoverse2 benchmark datasets. We also showcase that MapUnveiler significantly boosts the construction of vectorized map elements in challenging settings, achieving +10.7% mAP increase for longer perception ranges and +16.7% mAP improvement in heavily occluded driving road scenes. #HDMap #MapConstruction Paper link: https://lnkd.in/g-krc_TP [2] Unified Domain Generalization and Adaptation for Multi-View 3D Object Detection (in collaboration with Korea University / 고려대학교) Gyusam Chang*, Jiwon Lee*, Donghyun Kim, Jinkyu Kim, Dongwook Lee, Daehyun Ji, Sujin Jang, and Sangpil Kim Typical supervised learning approaches face challenges in achieving satisfactory adaptation toward unseen and unlabeled target datasets (i.e., direct transfer) due to the inevitable geometric misalignment between the source and target domains. In practice, we also encounter constraints on resources for training models and collecting annotations for the successful deployment of 3D object detectors. In this paper, we propose Unified Domain Generalization and Adaptation (UDGA), a practical solution to mitigate those drawbacks. Our framework enables stable detection performance in both source and target domains, effectively bridging inevitable domain gaps, while demanding fewer annotations. We demonstrate the robustness of UDGA with large-scale benchmarks: nuScenes, Lyft, and Waymo, where our framework outperforms the current state-of-the-art methods. #DomainGeneralization #DomainAdaptation #3DObjectDetection Paper link: https://lnkd.in/ge--ezQi

87

1 Comment

Join us in Irvine on May 15th at 6:30 PM for our latest OC ACM talk, "Understanding Life via Computational Bioinformatics" presented by Dr. Shawn X. Wang, a distinguished professor of computer science at Cal State Fullerton. For more info or to RSVP click here: https://lnkd.in/gX72Ss_g Dr. Shawn X. Wang will provide an insightful exploration of computational bioinformatics, a field dedicated to extracting, organizing, and interpreting biological data to uncover new insights. Divided into two parts, the presentation will first highlight the transformative impact of computational bioinformatics research on enhancing quality of life and deepening our understanding of biology. In the second segment, Dr. Wang will offer a concise overview of his research endeavors, focusing on innovative algorithms and data analysis techniques applied to protein 3D structures, microarrays, and DNA sequence comparisons. Join us for a cohesive journey into the intersection of computation and biology. #acm #compsci #bioinformatics #techtalk #irvine #calstatefullerton #science #innovation

1

The Transformer Series: This video explains the concept of positional encoding in the transformer architecture. In our previous videos, we looked at Skip connections, scaled dot product attention and multi-head attention. Now, let’s shift our focus to another important aspect of the transformer architecture: Positional Encoding. Positional encoding is crucial for transformers to understand the order of input data when looking at sequential data, such as the usual english sentences where the order of the words is important. Positional encoding adds unique markers to each input token to provide context of its position in the sequence. This method allows the model to recognize patterns and relationships in data that depend on the sequence order—something essential in tasks like language understanding or time series analysis. Take a look at this video to discover how positional encoding enables transformers to map out complex underlying patterns in sequence data. https://lnkd.in/g4bi4_A3 #transformers #deeplearning #ai #ml #nlp

59

Hey everyone, We’re excited to announce the release of our latest AI visual inference samples (https://lnkd.in/eJ6A96px). The highlight of this release is the introduction of heterogeneous pipelines that leverage Intel® Xeon® AI capabilities in combination with hardware media processing. Here’s how it works (see illustration): -       take Intel® Xeon® Processors equipped with Intel® discrete GPU card(s) -       execute decoding and preprocessing on HW accelerated media engines from Intel® Data Center GPU Flex Series (or Intel® Arc™) -       run AI inference on Intel® Xeon® CPUs with AMX (Intel® Advanced Matrix Extensions) instructions support. This approach allows you to fully utilize media GPU engines, achieving significant speed-ups compared to GPU-only solutions. We particularly recommend the Intel® Data Center GPU Flex 140 Series, as it features two GPUs per card and a total of four hardware media engines. This setup provides better balancing with Intel® Xeon® Processors. The number of GPU cards per host will depend on your Intel® Xeon® CPU model. High-end versions may require several cards to fully balance AI capabilities for media analytics models like Resnet50. All new heterogeneous pipelines have the postfix "GPU_CPU." You can find them along with others in the folder samples/openvino. #IAmIntel #openvino #intel #ai #deeplearning

73