Vasudev Lal

There is often an unspoken assumption by ML researchers that all we need to make progress and model every aspect of our world is data. We do not ascribe to this in my lab. In fact, we believe that data will never be enough. Our experiences with the nuances and complexities of scientific problems have informed us to the insufficiency of data to capture continuous physical processes, which afterall govern our biological and physical world. An example of this is this series of two papers led by my wonderful PhD student, Anowarul Kabir and advanced by a precious multi-year collaboration of my lab with Los Alamos National Lab: Anowarul Kabir, Manish Bhattarai, Kim Rasmussen, Amarda Shehu, Anny Usheva, Alan R Bishop, and Boian S Alexandrov. Examining DNA Breathing with pyDNA-EPBD. Bioinformatics 39(11):btad699, 2023. https://lnkd.in/gZaHE4hS Anowarul Kabir, Manish Bhattarai, Selma Peterson, Yonatan Najman-Licht, Kim Ø Rasmussen, Amarda Shehu, Alan R Bishop, Boian Alexandrov, Anny Usheva. DNA Breathing Integration with Deep learning Foundational Model Advances Genome-wide Binding Prediction of Human Transcription Factors. Nucleic Acids Research: gkae783, 2024. https://lnkd.in/gfTTqcPy Our goal: advance an exceptionally challenging problem in molecular biology, prediction of transcription factor binding sites. Our first step: capture the underlying physics that is missing in the data. Our second step: integrate that now with the data we have in a foundation model for predicting transcription factor binding sites. Performance improves. Most importantly, when we look at the sequence motifs that constitute "signatures" of what makes a transcription factor binding site, we obtain answers. All in the open, nothing opaque.

33

1 Comment

Check out our new work on LLM reasoning! A fine-tuning-free approach that can reach the reasoning level of 300B+ closed-resource LLMs using only ~10B open-source LLMs #LLM #Reasoning #M*

13

Come see this talk, which I view as the culmination of several years of discussions and co-teaching with George Fairbanks. It's a bit of an overview, a bit of an explainer, a bit of a how-to, but mostly a sneaky way to make some important vocabulary from design become load-bearing in how you think about software testing.

86

1 Comment

Very insightful work by Anthropic’s interpretability team. And an amazing paper, with outstanding writing and figures. The idea is very simple: interpret LLMs by leveraging sparse autoencoders as surrogate models of the MLP of transformer blocks, which allow one to disambiguate the superposition of features captured by a single neuron. A simple idea, but a very careful and complex execution, as it is often the case in our line of work. The paper goes into many details and provide a large array of insights, although the gist of the implementation remains obfuscated due to the closed source nature of Claude. Too bad, because this is the kind of work that we need to better understand and eventually trust LLMs. This is demonstrated by the authors in the section ‘Influence on Behavior’, where they show that clamping some features to either high or low value during inference is “remarkably effective at modifying model outputs in specific, interpretable ways”. Hopefully this kind of work is going to be replicated and generalized to open-weights models, such that we have new ways to steer their behavior. https://lnkd.in/eVym7f_f #interpretability #xai #explainableai #steerableai #anthropic #claude #anthropic

3

Here's the technical breakdown of the AI Penalty Challenge authored by Alok Pattani (check out this video if you haven't seen the demo yet: https://lnkd.in/gsD3xjJ2) Key Takeaways: ☁️ Firestore for triggering slow-mo video capture on Android phones & uploading to Google Cloud via Firebase Storage. 🧠 Cloud AutoML for training custom object detectors (ball & goal tracking) to calculate speed & accuracy. 🗣️ Gemini Pro 1.5 on Vertex AI for personalized coaching feedback based on multiple data points. 🎨 Imagen on Vertex AI for instant, prompt-driven image editing for cool player pics. More technical insights from the project to come... #AIChallenge #MachineLearning #SportsTech #GoogleCloud #footballNOTsoccer

89

5 Comments

I appreciate the insight from Prof. Shashua to highlight the pos and cons of E2E and abstraction. I think that Tesla took the route of Data Center + Data Engine + E2E on the edge while minimizing engineers' intervention. It is not necessarily the best route for the other competitors. In other market, engineers' intervention in the design could be greater or better than the data engine at that time.

2

🤗 Transformers now supports speculative decoding for ANY model - bringing ~2x acceleration to 140k+ LLMs on the Hub, without accuracy loss 🤯 Introducing Universal Assisted Generation -- a new method from Intel Labs and Hugging Face that expands assisted generation to support target / assistant model pairs across ANY model family 🔥 > Update to the latest 🤗 Transformers version (see first comment for pipeline usage below) > Choose a target model from one of 140k+ LLMs or Mixture of Experts models from the 🤗 Hub > Select a lightweight assistant model such as Qwen2.5-0.5B or Vicuna-68m to generate draft tokens > Accelerate target model inference by ~2x 🔥 > Tokenizer mismatches are identified and corrected behind the scenes, with minimal overhead Check out the blog for more details: https://lnkd.in/dYwgMgWV Grateful for the collab with Oren Pereg, Moshe Berchansky, Jonathan Mamou, João Gante, Lewis Tunstall, Nadav T. and Moshe Wasserblat

357

15 Comments

These findings motivate our design of a novel framework to investigate hidden representations in LLMs with LLMs, which we call Patchscopes. The key idea behind this framework is to use LLMs to provide natural language explanations of their own internal hidden representations. Patchscopes unifies and extends a broad range of existing interpretability techniques, and it enables answering questions that were difficult or impossible before. For example, it offers insights into how an LLM's hidden representations capture nuances of meaning in the model's input, making it easier to fix certain types of reasoning errors. While we initially focus the application of Patchscopes to the natural language domain and the autoregressive Transformer model family, its potential applications are broader. For example, we are excited about its applications to detection and correction of model hallucinations, the exploration of multimodal (image and text) representations, and the investigation of how models build their predictions in more complex scenarios. Google DeepMind

9

I remember when this all started. Faris - invited me to catch up at our favorite spot around Blok M. I had no idea he’d just wrapped up a procedure at Pondok Indah Hospital! (He really knows how to add drama to our catch-ups.) We’d been on separate paths and projects since "the winding down project", but there he was, reminding me of our dream to ride this AI wave together again. It was a plan we’d hatched with Sahal Zain & Gurnoor Dhillon when the world of AI was still mostly theoretical and jargon-heavy. Somehow, the conversation continued with Gurnoor Dhillon while he was waiting for his meeting with someone who would eventually become one of CalvinBall’s key early adopters. It felt like everything was coming together, like a series of seemingly random dots connecting to form a bigger picture. Seeing how far we’ve come as a team at CalvinBall Technologies is truly exciting. What started as theoretical discussions has now evolved into practical Gen AI solutions that are solving everyday problems and hold endless possibilities to shape the industry. 👨‍🚀 👩‍🚀 Here’s to pushing boundaries and achieving new milestones together! Gurnoor Dhillon, Sandeep Ramesh, Faris -, Sahal Zain, Eva Zuliya, Ratu Nadiah Khairunnisa 🚀 #GenAI #AIInnovation #TechJourney #ArtificialIntelligence #TechTransformation #CalvinballTech #Calvinballers

55

2 Comments

SEMICON WEST is on fire 24x7 Tuesday afternoon continued the focus on Front-End Manufacturing. Great session on wafer map visual analytics with Janhavi Giri, Intel, using self-supervised learning for image analysis, detecting failures and optimizing manufacturing processes. Naor Alfassi, Applied Materials, spoke about ML-guided Electron Beam review for wafermap image classification and defect detection. Janhavi Giri, PhD Naor Alfassi Sessions on virtual metrology from Srividya Jayaram, Siemens, on high product mix; and Mike Yim, Gauss Labs, on high-volume, from Lab to Fab. In our experience, virtual metrology has led to massive time savings in product testing.  Srividya Jayaram Mike Young-Han Kim One very fun and topical presentation was from Christoher Nguyen, Aitomatic, and Daisuke Oku, Tokyo Electron - announcing SemiKong, a new Open-Source Semiconductor LLM, jointly developed by Tokyo Electron, Aitomatic, FPT AI Center and the AI Alliance. This seems to be based on Meta's Llama3 with additional HTM context. Christopher Cuong T. Nguyen Daisuke Oku I just returned from visiting our HTM customers in Japan and Korea, where we showed our Spotfire Copilot running in Japanese and Korean .. and this presentation on SemiKong was a joy to see!! Feel free to check out my latest blog on the Japan visit including a Spotfire Seminar with our partner NTT Com and customers including Texas Instruments, Tokyo Electron, Toshiba, Sony and many others. MOC Blog on Spotfire Seminar in Tokyo, Japan https://lnkd.in/gAeSZSSC   Wednesday sessions in Smart Manufacturing focused on how AI/ML can optimize back-end semiconductor manufacturing process eg OEE, predictive and condition-based maintenance.. We are currently revamping our Spotfire OEE solution with Spofire Copilot, so this is quite timely. Shawn Sarbacker, Kulicke and Soffa,  gave a nice presentation on Semiconductor Assembly.  Shawn Sarbacker I really enjoyed the session on Zero Defect Manufacturing from Prasad Bachiraju, Onto Innovation. Totally aligned with our Spotfire work on visual data science for defect analysis .. interesting backend data integration .. we’ve been working on data federation and libraries of virtualization views for that part of the data and analytics flow.  Prasad Bachiraju I also liked the IBM WatsonX GenAI session from Megan Arp and Francois Bernard - many similarities to our Spotfire Copilot e.g. they refer to the “Watson Orchestrator” in an analogous architecture to our Spofire Copilot Orchestrator. Their LLM is able to predict user intent when a technician reviews a ticket (i.e., query vs. task). Tasks are handled through custom and prebuilt skills training, while queries are handled through RAG. Megan Arp Francois Bernard Spotfire #VisualDataScience https://lnkd.in/eSeVcE5h https://lnkd.in/ecWq3PSH

62

4 Comments

High Performance being in my expertise, I love these kind of news: An eight-member research team from Australian and American institutions has been awarded the 2024 ACM Gordon Bell Prize for their project, "Breaking the Million-Electron and 1 E FLOP/s Barriers: Biomolecular-Scale Ab Initio Molecular Dynamics Using MP2 Potentials." Using their algorithmic innovations on a powerful exascale computer, the team achieved a record-breaking performance of simulating more than one million electrons for a computational chemistry application and scaling their algorithm to an exaflop per second, resulting in a simulation 1,000 times larger in system-size than the existing state-of-the-art and 1,000 times faster than any previous model. Paper link: https://lnkd.in/dR96bNJn The ACM Gordon Bell Prize is awarded each year to recognize outstanding achievement in high-performance computing. The purpose of this recognition is to track the progress of parallel computing and rewards innovation in applying high-performance computing to challenges in science, engineering, and large-scale data analytics.

2

This is also the XPU using scenario In my mind too. “Whisper on NPU, LLMs on iGPU, document embeddings on CPU! Full use of #AIPC through #OpenVINO to implement scenarios like chat with documents and voice guided assistant that generates code. Everything on our new Lunar Lake platform that was just announced.”

9

The 1st deadline for the NVIDIA Academic Grant program is in 1 week. We're accepting proposals for #GenAI #LLMs and Simulation & Modeling. You mustbe a full-time faculty member at an accredited academic institution that awards research degrees to PhD students. NVIDIA will be funding proposals for innovative work related to generative AI and large language models (LLMs), including multi-modal models and content generation, such as diffusion models. Projects should incorporate pretrained models from ai.nvidia.com or use NIMs, Modulus, TensorRT(-LLM), NeMo, NeMo Retriever, NeMo Data Curator, BioNemo, NeMo Guardrails, or Isaac Sim/Lab. Gen AI & LLM Proposals should be attached to one of more of the following themes: Techniques for Training, Customizing, and Accelerating GenAI Exploring New Approaches and Architectures for LLMs and GenAI or Foundational Models and Scientific Applications Using GenAI Simulation Proposals should be attached to one of more of the following themes: Simulation and modeling that incorporate Modulus, Omniverse, HPC SDK, cuNumeric, cuQuantum, or CUDA Quantum. Proposals should be attached to one of more of the following themes: Computational Fluid Dynamics Physics-Informed Machine Learning Quantum Computing Scientific Simulation Selected Principal Investigators (PIs) may receive Up to 32,000 A100 40GB hours or equivalent OR up to 8 RTX 6000 Ada

5

Arm's partnership with the PyTorch team at Meta is transforming the AI landscape. Looking to run your generative AI workloads faster at the edge? Read on! Arm's teamed up with the PyTorch team at Meta to optimize AI performance through the ExecuTorch framework, which is now available in Beta. This brings instant enhancements to your AI workloads running on the Arm compute platform. Combine this with AI at Meta's new quantized Llama 3.2 1B and 3B models, and you get Gen AI models with reduced memory footprint, higher levels of accuracy and greater performance.

19

Hallucinations in LLMs are inevitable, no seriously! Josh Fritzsche and I have worked hard at Voze to find a way to tune, prompt and even plead 🙏 with LLMs to prevent them from the occasional 🍄 hallucination. It seemed like the better we have done the harder the edge cases have been to solve and eventually we realized that they tend to frequently happen when LLMs predict or make anticipations on "nothing". I remember at one point nearly going insane with frustration because when LLMs had a hallucination, they were EPIC! 😱 Then we pushed through it and benefited for the effort! How did we solve this common problem? We didn't (Not completely), we got better at identifying the issues as they happened. Once we identified in the data we were able to start looking for patterns. The same catalysts that cause it to happen once if left unchanged caused it over and over again. Our solution is astoundingly simple. Once you know that it's likely to happen..... let it! Accept it will 🍄, but also be preemptive in not focusing on it and use alternative data sources, generators or providers. Change the conditions that cause it but continue to monitor it as it happens. As Dan Caffee is fond of saying to me on a regular bases. "Dusty we need the system to be adaptive and provide mechanisms to learn from it's past." Well he is right! All we needed was data and the people that contribute to it's quality. Huge props 👏 to strong support from Kathy and Janelles audit teams. We were armed with a knowledge and a desire to learn from past mistakes. Armed with confidence that we could correct these problems we could focus on the data to prevent their impact going forward. When doing millions of LLM (Generative AI) calls regularly Josh and I have learned that hallucinations will happen. We also know exactly how we will handle and prevent it going forward! Failure is not doing anything and accepting defeat.

4

2 Comments

Interested in learning more about what powers #AppleIntelligence? This paper describes in detail the Language Models that power these features. Please reach out if you want to join us in building incredible features powered by generative models that bring surprise and delight to millions of users! https://lnkd.in/ghZxHD47

48

1 Comment

Thanks Hamdi Amroun, PhD - this is a great paper to share. In the real world business context we often need AI to reason over large pieces of text, and often you need/want multi-document summarization (MDS). 'When testing five LLMs with benchmarks using news and conversation datasets, they found up to 75% of content in MDS summaries was hallucinated, with notable increases towards summaries' end. Alarmingly, even for non-existent topics, models like GPT-3.5-turbo and GPT-4 generated fabricated content 79% and 44% of the time, respectively. Analysis of 700+ generated insights showed that hallucinations often arose from failures to follow instructions or overly generic content.' I agree with Hamdi's sentiments that despite improvements in post-processing techniques, robust solutions are urgently needed to reduce hallucinations, as inaccurate summaries can lead to business risks (notably misinformation and misrepresentation). I would add that there is still a lot of utility in LLMs/Multimodal models but we need to be vigilant - this is not about AI malevolence or intentional deception. LLMs do not possess consciousness or intent; they generate content based on patterns in data they were trained on.

6

1 Comment

Witnessed the meeting of 2 great Tech giants- Jensen Huang and Mukesh Ambani, and keenly heard their profound conversation @Nvidia India Summit, Jio World Convention Centre, Mumbai- 24th Oct 2024- 1. Jensen Huang (Nvidia)- * There is a Techtonic shift in computing with AI. * AI factories and AI cloud featured C-DAC. * If you are good at one thing, it gives you the confidence to do so may other things! Self discipline and humility are so important to reach the success that Jensen has reached.. * INDIA and NVIDIA are rhyming. 2. Mukesh Ambani (Reliance Industries Limited) * VIDYA (rhyming with NVIDIA)- knowledge- revolutionizes intelligence. And this brings Lakshmi- prosperity! * AI- Aspirational India will be Innovation hub of the world. * India has the best infrastructure connectivity in the world- came from Number 158 to Number 1 in the world in 8 years, generated16 exa bytes of data. 15 cents a GB in India vs 4 dollars a GB in US. * India has Raw gene power== Youth in India!! * We will not only export CEOs to the world but make CEOs in India!!! * Make a more Equal world, a more prosperous Global South. * India will start with not just the best but the "Absolute Best". * Green power. India will surprise the world with Intelligence in a couple of years. * India has a Leader who believes in Vision and Execution. Quoted PM Modi- India should not export data to import intelligence. India has Data infra and users. AI has power to elevate the whole population in the world. * Llama3- Open source model from Mark Zuckerberg is well appreciated. Will help build useful solutions * Wished everyone a Happy Diwali too! Listen to the conversation-

12