Majid Mirbagheri

I'm delighted to share that our #AutonomousDriving team @Samsung SAIT have two papers accepted to NeurIPS 2024! Congrats to the authors, and thanks for the great support from our incredible team. #NeurIPS #NeurIPS2024 [1] Unveiling the Hidden: Online Vectorized HD Map Construction with Clip-Level Token Interaction and Propagation Nayeon Kim*, Hongje Seong*, Daehyun Ji, and Sujin Jang In this paper, we introduce a novel paradigm of clip-level vectorized HD map construction, MapUnveiler, which explicitly unveils the occluded map elements within a clip input by relating dense image representations with efficient clip tokens. Our extensive experiments demonstrate that MapUnveiler acheives state-of-the-art performance on both the nuScenes and Argoverse2 benchmark datasets. We also showcase that MapUnveiler significantly boosts the construction of vectorized map elements in challenging settings, achieving +10.7% mAP increase for longer perception ranges and +16.7% mAP improvement in heavily occluded driving road scenes. #HDMap #MapConstruction Paper link: https://lnkd.in/g-krc_TP [2] Unified Domain Generalization and Adaptation for Multi-View 3D Object Detection (in collaboration with Korea University / 고려대학교) Gyusam Chang*, Jiwon Lee*, Donghyun Kim, Jinkyu Kim, Dongwook Lee, Daehyun Ji, Sujin Jang, and Sangpil Kim Typical supervised learning approaches face challenges in achieving satisfactory adaptation toward unseen and unlabeled target datasets (i.e., direct transfer) due to the inevitable geometric misalignment between the source and target domains. In practice, we also encounter constraints on resources for training models and collecting annotations for the successful deployment of 3D object detectors. In this paper, we propose Unified Domain Generalization and Adaptation (UDGA), a practical solution to mitigate those drawbacks. Our framework enables stable detection performance in both source and target domains, effectively bridging inevitable domain gaps, while demanding fewer annotations. We demonstrate the robustness of UDGA with large-scale benchmarks: nuScenes, Lyft, and Waymo, where our framework outperforms the current state-of-the-art methods. #DomainGeneralization #DomainAdaptation #3DObjectDetection Paper link: https://lnkd.in/ge--ezQi

87

1 Comment

Back in May, I was invited by the local chapter of IEEE SSCS in Santa Clara to give a talk on how to use LLM and Gen AI for high-speed design. The video has been uploaded to YouTube. It is a good summary of how we are using LLM and AI for SI. https://lnkd.in/eUy2HDhB A few months later, Eric Schmidt gave a controversial talk on where he thought AI would progress in the next few years in a Stanford computer science class. He was a little too frank on telling students to steal TikTok IPs and why he thought Google would fail. https://lnkd.in/epMKkmvv Both talks seem to completely overlap on how to create domain knowledge dominance using advanced LLM. He gave examples on taking over TikTok and in chemical engineering, I was focusing on high-speed SI designs. But it can be applied to any other hardware design disciplines.  What we are pursuing, and he seems to suggest the same approach : 1) Use LLM with deep context window to scan through large volume of knowledge (text, video or sound), in my case for SI/PI we will be scanning large volumes of design specs for design domain knowledge 2) Use LLM agents to execute task based on the ingested knowledge (context to action), in my case, we will be initiating simulations based on design spec range and look for optimal solutions 3) Repeat the above with multiple chain of thoughts and store the observation to draw a conclusion, in my case, after scanning through the design specs and obtain optimal spacing for performance, generate routing and constraint rules 4) We also add an optimization step which is necessary for engineering  5) Improve the LLM agent performance using an adversarial agent to critique the LLM agent's performance 6) We also add using fine-tuning and transfer learning to future proof the domain specific models. I apologize both are long talks ~1 hour.  I hope this will be a good starting point to discuss the use of LLM and Gen AI for SI/PI. Feedback and comments are most welcome.

13

2 Comments

Current language-based agents struggle with long-horizon tasks and lack the ability to learn reusable task workflows from past experiences. This recent paper on arXiv aims to introduce "Agent Memory Workflow (AWM)", a method that can induce commonly reused task routines, called "workflows", and selectively provide them to agents to guide their future actions. This can help language model-based agents solve complex real-world tasks, such as web navigation. 🧮 Methodology: The authors propose two ways to implement AWM: 1. Offline AWM: Workflows are induced from available training examples and integrated into the agent's memory before solving test tasks. 2. Online AWM: Workflows are induced from the agent's own predictions on test tasks and continuously added to the agent's memory during inference. 🎯 Results and findings: This approach has shown significant improvements in web navigation tasks, covering 1000+ tasks from 200+ domains, with: 1. 24.6% relative success rate increase on the Mind2Web benchmark 2. 51.1% relative success rate increase on the WebArena benchmark AWM also demonstrates strong generalization abilities, outperforming baselines by 8.9-14.0 absolute points on cross-website and cross-domain evaluations. 🍂 Limitations and future work: While workflows generally improve agent performance, there are still challenges in identifying when to diverge from the prescribed workflow guidelines. Future work could explore techniques to grant agents more dynamic access to the environment state during workflow execution to better adapt to changes. Link to the paper: https://lnkd.in/gWZ_RbwH #Agents #GenAI

13

In-Context Learning at Extreme Scales: My Thoughts As large language models (LLMs) continue to expand their context capacities, the domain of in-context learning (ICL) at extreme scales has piqued my interest. A recent paper by researchers at Carnegie Mellon University and Tel Aviv University provides a detailed exploration into this area, unveiling both impressive capabilities and notable challenges. 🔍 Key Insights: Performance Improvement: The study demonstrates that ICL can improve significantly by integrating hundreds to thousands of examples into the context window—far beyond traditional few-shot settings. This is particularly impactful in complex tasks like 77-way banking intent classification, where long-context ICL matches or surpasses traditional finetuning. Input Sensitivity: Long-context ICL shows reduced sensitivity to input shuffling compared to shorter contexts. However, organizing examples by label dramatically reduces performance, emphasizing the need for diverse context configurations. Source of Gains: Performance gains primarily arise from accessing more relevant examples, rather than aggregating knowledge across many examples during encoding. 📉 Challenges & Drawbacks: Sensitivity to Example Order: Despite reduced sensitivity, the performance still varies with input order, indicating potential instability in practical scenarios. Performance Saturation: The models often do not fully leverage their extended context capacities, showing performance saturation before reaching the maximum context length. Computational Cost: The increased computational cost associated with using large context windows could limit practical deployment, particularly in resource-constrained environments. Efficiency Issues: There is a noted inefficiency in how models utilize the encoded information within large datasets, suggesting that current architectures may need refinement to better use available data. 🚀 Future Directions: The paper suggests several exciting avenues for future research, including optimizing the trade-offs between computational costs and performance, exploring model adaptation strategies, and refining ICL approaches for specific applications or datasets. As we push the boundaries of what LLMs can achieve, effectively leveraging their full context could unlock new capabilities. This area is ripe for further exploration and could lead to major advancements in how we deploy AI. For those at the cutting edge of language model development, I highly recommend this paper for its thorough analysis and forward-looking perspective. Check out the full work at https://lnkd.in/d7AmGh4y. What are your thoughts on the future of in-context learning at such scales and the challenges discussed? #AI #MachineLearning #DeepLearning #LanguageModels #InContextLearning

17

The Transformer Series: This video explains the concept of positional encoding in the transformer architecture. In our previous videos, we looked at Skip connections, scaled dot product attention and multi-head attention. Now, let’s shift our focus to another important aspect of the transformer architecture: Positional Encoding. Positional encoding is crucial for transformers to understand the order of input data when looking at sequential data, such as the usual english sentences where the order of the words is important. Positional encoding adds unique markers to each input token to provide context of its position in the sequence. This method allows the model to recognize patterns and relationships in data that depend on the sequence order—something essential in tasks like language understanding or time series analysis. Take a look at this video to discover how positional encoding enables transformers to map out complex underlying patterns in sequence data. https://lnkd.in/g4bi4_A3 #transformers #deeplearning #ai #ml #nlp

59

As I delved into image processing using deep neural networks, I pondered the mechanism by which neural networks address the 2D spatial correlations inherent in images. Initially, the early layers of these networks appeared to map pixel intensities to individual neurons in the input layers, seemingly neglecting the 2D spatial relationships. This approach raised concerns about the effectiveness of traditional neural networks in capturing spatial dependencies. However, my understanding and concerns were alleviated upon discovering Convolutional Neural Networks (CNNs), which are specifically designed to account for and exploit these spatial correlations in images.

8

Have you heard about tiny AI accelerators (smaller than 1cm x 1cm) that can even run on microcontrollers? I’m excited to share that I’ll be attending #NeurIPS2024 to present Nokia Bell Labs' initiative on building collaborative on-device AI systems: “DEX: Data Channel Extension for Efficient CNN Inference on Tiny AI Accelerators.” This work introduces an innovative approach to improving AI model accuracy without increasing latency, leveraging intelligent data channel extensions on tiny AI accelerators. Huge thanks to Taesik Gong and Fahim Kawsar for their invaluable collaboration! This marks the beginning of our vision to enable multi-device, multi-modal, and multi-sensory computing on the body. Stay tuned for more exciting projects in this direction! * When: Wednesday, Dec. 11, 4:30 PM – 7:30 PM * Where: East Exhibit Hall A-C, Booth 1405 * arXiv: https://lnkd.in/emP4_5vg * Code: https://lnkd.in/ecn5yTCB * Broader overview of our department research: https://lnkd.in/enbW6fnW Summary: Tiny AI accelerators enhance TinyML by improving processing power, but their limited memory often reduces input resolution, negatively impacting accuracy. To address this, we introduce DEX—a method that extends input channels by incorporating spatial information through patch-wise sampling and stacking. DEX leverages unused processors and memory for parallel execution, maintaining inference latency while improving accuracy by an average of 3.5 percentage points.

60

4 Comments

Kudos to #Composable_kernel team and please check out the AMD engineer Haocong Wang presents the ROCm Composable Kernel library which allows to write high-performance kernels for machine learning with HIP C++. #Composable_kernel #ROCm

39

1 Comment

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts https://lnkd.in/gb78Y4bn In the latest foray into the world of multimodal language models, researchers have unveiled an intriguing new approach they've dubbed "CuMo." This novel technique aims to harness the power of Mixture-of-Experts (MoE) architectures, incorporating Co-upcycled Top-K sparsely-gated MoE blocks into the vision encoder and the multi-layer perceptron connector of multimodal LLMs. The incorporation of these MoE components is a bold move, promising to enhance the capabilities of multimodal LLMs in a significant way. However, the researchers didn't stop there. They've also adopted a three-stage training approach that employs auxiliary losses to help stabilize the training process and ensure a balanced distribution of workload across the expert modules. One of the most noteworthy aspects of this project is that CuMo has been exclusively trained on open-source datasets, a commendable decision that promotes transparency and accessibility in the field of AI research. Overall, this project presents an exciting exploration of MoE architectures in the context of multimodal language models. While the true impact of CuMo remains to be seen, the innovative approaches employed and the promising early results make this a development worth keeping an eye on in the rapidly evolving field of AI.

3

Excited to share that our latest research paper, titled "Run-Time Adaptation of Neural Beamforming for Robust Speech Dereverberation and Denoising," has been accepted to APSIPA ASC 2024 and is now available on arXiv! Abstract: We tackle the challenges of speech enhancement for real-time automatic speech recognition (ASR) in noisy, reverberant environments. While neural beamforming has been a go-to approach, it often struggles under mismatched conditions. Our work introduces a run-time adaptation technique for neural networks using pseudo ground-truth data generated by blind dereverberation and separation methods, such as Weighted Prediction Error (WPE) and Fast Multichannel Nonnegative Matrix Factorization (FastMNMF). To improve robustness, we introduce an adaptive beamforming method: Weighted Power Minimization Distortionless Response (WPD) beamforming. By unifying WPE and Minimum Power Distortionless Response (MPDR), our approach integrates dereverberation and denoising capabilities, enabling real-time fine-tuning. We evaluated this method across diverse settings, exploring the impact of run-time adaptation in scenarios with multiple speakers, varying reverberation times, and a range of signal-to-noise ratios (SNRs). Check out the full paper on arXiv to learn more about our methodology, experiments, and insights: https://lnkd.in/gq5y_JqH Proud to be sharing this work with the community at APSIPA ASC 2024! #SpeechRecognition #ASR #MachineLearning #NeuralNetworks #Beamforming #DNN #Research #APSIPA2024

20

LLMs achieve adult human performance on higher-order theory of mind tasks https://lnkd.in/g-A_JFr7 The paper examines the ability of large language models (LLMs) to develop higher-order theory of mind (ToM) - the human capacity to recursively reason about multiple mental and emotional states (e.g. "I think that you believe that she knows"). The authors introduce a new handwritten test suite called "Multi-Order Theory of Mind Q&A" to evaluate five LLMs against a newly gathered adult human benchmark on ToM tasks. The results show that GPT-4 and Flan-PaLM reach adult-level and near adult-level overall performance on ToM tasks, respectively. Notably, GPT-4 even exceeds adult human performance on 6th order ToM inferences. The findings suggest an interplay between model size and finetuning in the realization of ToM abilities in LLMs. The best performing models appear to have developed a generalized capacity for theory of mind reasoning. Given the critical role higher-order ToM plays in human cooperative and competitive behaviors, these results have significant implications for user-facing applications of large language models that need to understand recursive mental state reasoning. In summary, state-of-the-art LLMs like GPT-4 demonstrate human-level or better ability to make higher-order theory of mind inferences, an important milestone with real-world implications.

21

RAG of 2024 (ColbertV2 Reranking) !!! I've just published an in-depth guide on how to build an efficient local semantic search system using the innovative technologies of ColBERTv2, DSPy, and Qdrant . If you're interested in advancing your search capabilities with the latest AI technology, this article is for you. 🔍 Discover how the deep contextual understanding of ColBERTv2 and the dynamic optimization capabilities of DSPy can transform your data retrieval processes. Plus, learn about integrating Qdrant for scalable and precise search results. 📖 Check out the full article here: https://lnkd.in/gi_mQNH8 #SemanticSearch #AI #MachineLearning #DataScience #ColBERTv2 #DSPy #Qdrant #TechInnovation

20

2 Comments

🤗 Transformers now supports speculative decoding for ANY model - bringing ~2x acceleration to 140k+ LLMs on the Hub, without accuracy loss 🤯 Introducing Universal Assisted Generation -- a new method from Intel Labs and Hugging Face that expands assisted generation to support target / assistant model pairs across ANY model family 🔥 > Update to the latest 🤗 Transformers version (see first comment for pipeline usage below) > Choose a target model from one of 140k+ LLMs or Mixture of Experts models from the 🤗 Hub > Select a lightweight assistant model such as Qwen2.5-0.5B or Vicuna-68m to generate draft tokens > Accelerate target model inference by ~2x 🔥 > Tokenizer mismatches are identified and corrected behind the scenes, with minimal overhead Check out the blog for more details: https://lnkd.in/dYwgMgWV Grateful for the collab with Oren Pereg, Moshe Berchansky, Jonathan Mamou, João Gante, Lewis Tunstall, Nadav T. and Moshe Wasserblat

357

15 Comments

This new article discusses our efforts in delivering tensor parallelism, weight-only quantization, vLLM integration, and the RBLN model zoo. It is definitely worth reading! 👉 https://lnkd.in/gRnG8_pR #TensorParallelism #WeightOnlyQuantization #vLLM #ModelZoo

84

We are still hardly scratching the surface of what is possible with LLMs.

30

2 Comments

I've been playing with matrix multiplication on GPUs, using a Hilbert curve to improve L2 cache utilization, and seeing if I can get non-embarrassing performance out of it. Seems this approach can outperform Nvidia's cuBLAS for matrix sizes of 16384x16384 and above. Blog post here with open source code on GitHub for anyone who'd like to mess with it too. #gpu #cuda #cplusplus https://lnkd.in/e4QvmSVR

61

4 Comments

A substantial piece of work from Apple led by Iman Mirzadeh, keivan alizadeh vahid, Oncel Tuzel, Samy Bengio, Mehrdad Farajtabar and Hooman Shahrokhi supports the thesis that transformer-based LLMs do not reason--they perform pattern-matching. The team from  also provides a great benchmarking tool--GSM-Symbolic for assessing mathematical and formal reasoning. This supports my anecdotal findings when I evaluated o1 https://lnkd.in/ec72fecb as well as my thoughts on the incompleteness of our assessments of AGI https://lnkd.in/ehBT3HB3

16

2 Comments

An interesting topic

17