Tao Lin

Hope you had a chance to see our NVIDIA Holoscan for Media solution on the Monks booth, running real-time brand detection with minimal latency. Under the hood: - Fine-tuned YOLO v8 from the NVIDIA NGC Model Catalog using NVIDIA TAO Toolkit - Integrated with NVIDIA DeepStream SDK as part of the AI Pipeline within NVIDIA Holoscan for Media

44

2 Comments

The Notebook LM (Experimental) is available in Korea, despite the time difference from the US. Leveraging the power of Gemini 1.5 Pro for personal RAG conversations, this innovative tool creates an engaging experience by integrating personal data into AI interactions. Specifically, it summarizes data that individuals are interested in and makes it easy to ask questions, summarize, and make suggestions based on that data. The impact has been significant, with new users moving beyond simple interactions to engage with AI for in-depth content reviews and thoughtful discussions. We look forward to seeing LLM technology evolve to enable faster, more personalized AI interactions through local inference. #innovation #AI #LLM #personalizedAI

1

Using generative AI, NVIDIA operations built an AI Planner agent, developed on NVIDIA Inference Microservices (NIM). The agent leverages LLM, NeMo Retriever and CuOpt NIM to reduce re-planning time from hours to just seconds. https://nvda.ws/4bIfS2T

1

Experience PaliGemma, Google's newest multimodal model powered by NVIDIA NIM microservice and optimized by TensorRT-LLM, for generating informative text responses from text and visual inputs. Try the API now: https://lnkd.in/g8J3xW4K

1

Delve into a 5-min guide on how to deploy #NVIDIANIM with Docker, using a single optimized container. Read the tutorial > https://nvda.ws/3VpeE6V

210

3 Comments

🚀 Powering ML with Ray: Insights from Zhe Zhang's Tech Talk 🖥️ Last month, we had the privilege of hosting Zhe Zhang (Distinguished Engineer at NVIDIA and Ray expert) at Granica HQ in Mountain View 🙌 Zhe delivered a “no-BS”, thought-provoking talk that dove into a critical question: 💡 How do you command and utilize remote servers effectively for ML workloads? Here are the 🔑 takeaways from Zhe’s talk: 1️⃣ RPC is foundational: The concept of Remote Procedure Calls (RPCs) is the building block for distributed applications, enabling computation across remote servers. 2️⃣ The cluster spectrum: Frameworks like K8s, Ray, and Apache Spark allow us to scale workloads across clusters, each offering unique tradeoffs for ML practitioners. 3️⃣ Choosing the right framework: - 🏈 K8s shines for workloads resembling American football (quarterback passing to multiple runners). - ⚽ Ray excels when your workload mirrors soccer (everyone passing to everyone). - Apache Spark simplifies things—if your workload fits their expressive language, it "does the RPCs for you." We’re incredibly grateful to Zhe for sharing his practical, real-world perspective on ML infra—one shaped by his hands-on experience at NVIDIA and Anyscale. 🎉 The session was both insightful and approachable with a bunch of helpful Q&A, and his session is a must-watch for anyone navigating the challenges of scaling ML workloads 📖 Thanks again, Zhe, for bringing your expertise to Granica HQ! 🙏 👉 Check out Zhe’s blog (with the session recording inside) using the link in the comments. #MachineLearning #Infrastructure #DistributedComputing #MLPractitioners

23

1 Comment

12 techniques to reduce your LLM API bill and launch blazingly fast products Cost-efficiency and minimal latency are critical for deploying production-ready LLM applications.   Sahar Mor's article sheds light on 12 innovative techniques to slash your LLM API bills, with gems like Semantic Caching and Summarizing Conversations leading the charge. This isn't just about saving tokens; it's a strategic playbook for scaling smartly in an AI-dominated landscape.  #MachineLearning #AI #NLP #CostOptimization #Innovation #TechTrends https://lnkd.in/ec3qst-M

19

1 Comment

This is also the XPU using scenario In my mind too. “Whisper on NPU, LLMs on iGPU, document embeddings on CPU! Full use of #AIPC through #OpenVINO to implement scenarios like chat with documents and voice guided assistant that generates code. Everything on our new Lunar Lake platform that was just announced.”

9

Vara Bonthu and the team continue to produce incredible assets, driving forward the extraordinary partnership that empowers customers to scale Large Language Models (LLMs) more efficiently on Amazon EKS with NVIDIA's Inference Microservices (NIM) Today’s spotlight is on how this collaboration makes deploying Meta's Llama-3-8B model on Amazon EKS simpler and more effective. With dynamic scaling supported by Karpenter, the solution ensures optimal performance and resource efficiency. Key highlights include detailed guidance on deploying, managing, and scaling LLMs, plus insights into leveraging Prometheus and Grafana for complete observability. Whether your focus is low-latency or high-performance inference, this approach lays the foundation for scalable, powerful AI model serving. Check out the AWS blog to dive into the full details and see how Vara’s leadership is helping customers fully leverage the power of NVIDIA NIM on EKS. Abhishek Sawarkar

15

1 Comment

In this episode, Xiaodi Hou and Pete Bigelow discuss Bot Auto's journey, and our innovative approach to autonomous trucking. Xiaodi drops some knowledge on what it takes to make self-driving trucks profitable and his passion for music; from classical to Judas Priest. 🎼 🤘 🎸 Check out the full episode below and join us in redefining the future of transportation. Thank you Pete and Automotive News! #AutonomousVehicles #SelfDrivingTrucks #Innovation #Podcast #BotAuto

27

Dive into how KV cache early reuse, fine-grained blocks, and efficient eviction algorithms can supercharge TTFT speeds. Efficient KV cache use is key to improving model response, speeding up inference, and maximizing throughput. With TensorRT-LLM's advanced KV cache management features, developers can take inference performance to the next level.

1

Enterprises can achieve high levels of inference performance and take full advantage of hardware acceleration on Intel Corporation hardware by integrating the #opensource Triton Inference Server with the OpenVINO backend. Learn more about this powerful solution for deploying and serving #machinelearning models.  https://lnkd.in/gPNHq-JX

52

Learn why the NVIDIA Jetson AGX Orin is the leading platform for running transformer models like GPT-J, vision transformers, and stable diffusion at the edge.

8

1 Comment

"Building on its legacy in compiler technology, Skymizer has been dedicatedly developing its upcoming Large Language Model (LLM) accelerator, aptly named EdgeThought." "This software-hardware co-design IP exemplifies the company’s commitment to optimizing edge inferencing systems in terms of computation, memory usage, power efficiency, and cost." "EdgeThought leverages Skymizer’s advanced compiler technologies to tailor AI performance features specifically to edge computing environments, which are often constrained by power and space." "With EdgeThought, Skymizer introduces a cost-effective solution that allows for the deployment of sophisticated AI capabilities without the need for heavy infrastructure investments." "Skymizer’s move is a clear indication of the evolving landscape in AI chip design, where the integration of software and hardware through compiler-first strategies is becoming increasingly crucial." "The company is expected to showcase EdgeThought and its various applications at the upcoming tech conference, Computex Taipei, where it will delve deeper into the technical aspects and benefits of its innovative co-design approach." The above content is excerpted from Skymizer's official website. Come to meet us at Computex 2024!

1

https://lnkd.in/gSekNjEX One request I always get from academic researchers is the opportunity to understand and research on real-world product traces. The blog post from Phitchaya Mangpo Phothilimthana and Saurabh Kadekodi has some nice information on synthesized storage IO traces that we released today at Google. (The methodology is also quite cool, if you haven't seen it before, I would look at the Google ASPLOS'24 paper on Thesios.)

223

1 Comment