Lingyan Wang

Learn how an image-text multi-modality model can perform image classification, image retrieval, and image captioning #llms #llmops #image #opencv #chatgpt4 #gemini #aiml https://lnkd.in/gJFh9Ruf

1

🚀 𝗥𝗲𝗰𝗲𝗻𝘁 𝗛𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝘀 𝗶𝗻 𝗔𝗜 𝗥𝗲𝘀𝗲𝗮𝗿𝗰𝗵 🚀 𝗚𝗿𝗼𝗸𝗙𝗼𝗿𝗺𝗲𝗿: Graph Fourier Kolmogorov-Arnold Transformers Introducing a novel Graph Transformer network that surpasses conventional self-attention mechanisms. 1. Tackles limitations of existing GTs in modeling complex node label patterns. 2. Incorporates learnable activation functions in the graph spectrum. 3. Demonstrates superior performance on multiple node and graph classification datasets. Paper: https://lnkd.in/gDKKk-2x 𝗦𝗧𝗔𝗥 𝗔𝗧𝗧𝗘𝗡𝗧𝗜𝗢𝗡: Efficient LLM Inference Over Long Sequences Recent advancements in Large Language Models (LLMs) now handle contexts extending to millions of tokens. 1. New attention mechanism designed to address computational challenges during inference while preserving accuracy. 2. Opens up new applications, including repository-level code analysis, multi-document summarization, and large corpus retrieval. Paper: https://lnkd.in/gtj29ghz 𝗚𝗹𝗶𝘁𝗰𝗵 𝗧𝗼𝗸𝗲𝗻𝘀 𝗶𝗻 𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀: 1. Categorization Taxonomy and Effective Detection. 2. Addressing the issue of glitch tokens that lead to hallucinations and erratic behavior in LLMs. 3.Introduction of GlitchHunter, a novel clustering-based detection method. 4.Constructs a Token Embedding Graph (TEG) and uses clustering for efficient detection. 5.High precision and recall across multiple LLMs. Paper: https://lnkd.in/gTVr8jgK #MachineLearning #AIResearch #GraphTransformers #LLM #DeepLearning

4

I wanted to share some insights from a recent paper from Amazon Science in which authors tackled an important challenge in language modeling: minimizing sequence truncation. In language models, truncating input sequences might lead to a loss of crucial information, especially when the models are dealing with long-context tasks such as document summarization or story generation. The authors introduce a clever solution by leveraging a heuristic from the Best Fit Bin Packing problem, an NP-hard problem, to reorder input documents and reduce truncation. The proposed approach helps retain more information compared to other strategies by optimizing how sequences are packed into the model's context window. Authors show that the bin packing solution had biggest performance gain reading comprehension (+4.7%), natural-language inference (+9.3%), context following (+16.8%), and program synthesis. They also found that best-fit packing helped prevent closed-domain hallucination. Authors claim that while the experiments primarily focused on LLM pretraining, best-fit packing is broadly applicable to fine tuning as well. https://lnkd.in/gWquw3ud

28

3 Comments

We recently had Jon-Saad Falcon from Stanford AI Group give a talk at SambaNova Systems. He talked about his research ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. You can check out the talk at - https://lnkd.in/eXkBTb-j RAG systems are really hard to evaluate. Lot of hand holding required with annotated data or manual inspection. Excited to see this research and its outcome to help alleviate this problem. Key to this idea is a composite system using LLM as a judge. LLM judges are trained using synthetic data on domain queries/answer pairs. Its exciting to see how our customers use SambaNova Systems Platform to enable these workflows efficiently. Exceptional speed allows for fast synthetic data generation! And ability to host multiple models on a single system with extremely fast model switching allows for efficient low cost deployment. Try out 405B running at 120+ tokens/second at https://fast.snova.ai/ Want to enable your own such workflows by hosting fine-tuned checkpoints? Sign up here - https://lnkd.in/ecAsSDRH

61

🔮RAG + KG: Powerful Combination for Translation Multi-AI Agents Large language models (LLMs) have shown great promise in machine translation, but they still struggle with contextually dependent terms, such as new or domain-specific words. This leads to inconsistencies and errors that are difficult to address. Existing solutions often depend on manual identification of such terms, which is impractical given the complexity and evolving nature of language. While RetrievalAugmented Generation (RAG) could provide some assistance, its application to translation is limited by issues such as hallucinations from information overload. =================== In this paper, researchers propose CRAT, a novel multi-agent translation framework that: 》 leverages RAG and causality enhanced self-reflection to address these challenges. This framework consists of several specialized agents: ✱ the Unknown Terms Identification agent detects unknown terms within the context, ✱ the Knowledge Graph (KG) Constructor agent extracts relevant internal knowledge about these terms and retrieves bilingual information from external sources, ✱ the Causality enhanced Judge agent validates the accuracy of the information, ✱ Translator agent incorporates the refined information into the final output. =================== This automated process allows for more precise and consistent handling of key terms during translation. Their results show that CRAT significantly improves translation accuracy, particularly in handling context-sensitive terms and emerging vocabulary. --- If your company is looking for an AI consultant to accelerate business growth, or if you are looking to build your career in AI, send me a message and I can help

4

Spatial Reasoning did not ‘emerge’ spontaneously in Large Language Models (LLMs) the way so many reasoning capabilities did. Humans have specialized, highly capable spatial reasoning capabilities that LLMs have not replicated. But every subsequent release of the major models — GPT, Claude, Gemini- promises better multimedia support, and all will accept and try to use uploaded graphics along with texts. #llms #chatgpt4 #gemini #llmops #aiml #ml https://lnkd.in/dFSWKqKs

2

After reading Anthropic and OpenAI success stories with sparse autoencoders for interpretability of text, I was curious about its application to images... After a quick search, I found out that Hugo Fry wrote a blogpost in LessWrong: https://lnkd.in/dA33fWqE which also provides some playground! Of course part of the success is due to the ImageNet structure itself (as many papers out there) but still is an interesting exercise. It enables lots of experiments, and I already can think about practical usage for improving image search in marketplaces. #interpretability #image #CLIP

7

3 Comments

Retrieval Augmented Generation (RAG) is revolutionizing how large language models (LLMs) access and utilize external knowledge, leading to more accurate and timely responses. However, existing RAG methods often rely on complex iterative refinements or additional training steps to improve retrieval outcomes. Our latest research introduces a simpler yet more effective approach. Introducing Speculative RAG, a new RAG framework that harnesses the power of two LMs – a smaller specialist LM for efficient output draft generation and a larger generalist LM for accurate draft verification. By splitting retrieved documents into subsets and generating drafts in parallel, we reduce input length and mitigate potential biases. The result? State-of-the-art performance with significantly reduced latency! 🚀 Speculative RAG redefines the speed-accuracy tradeoff for RAG. It outperforms conventional RAG systems across multiple benchmarks, boosting accuracy by up to 12.97% while reducing latency by 51% on PubHealth. Our framework eliminates the need for complex refinements or additional training, making RAG more streamlined and scalable. Paper: https://lnkd.in/g_7iEa_u Authors: Zilong Wang, Zifeng Wang, Long Le, Huaixiu Steven Zheng, Swaroop Mishra, Vincent Perot, Yuwei Zhang, Anush Mattapalli, Ankur Taly, Jingbo Shang, Chen-Yu Lee, Tomas Pfister

46

In this episode, we discuss LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding by Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, Vikas Chandra. LongVU presents a spatiotemporal adaptive compression method for processing long videos using Multimodal Large Language Models, efficiently reducing redundancy while preserving important visual information. It employs techniques like cross-modal queries, DINOv2 features, and token reduction to manage spatial and temporal information. This approach shows superior performance on video understanding benchmarks, handling lengthy videos effectively and demonstrating scalability even in smaller models.

3

ArXiv alert! Interesting and recent applied research done by scientists in our Foundation Models team at Netflix Research - Sejoon Oh, Moumita Bhattacharya, and Yesu Feng - is available on ArXiv. Here we motivate the problem of predicting users' session intents and propose a solution for it using Hierarchical Multi-Task Learning. Not only do we get good predictions of the (proxy) intents but also show that the architecture improves the foundation model. https://lnkd.in/gEGQRhg9 Hope you all like it. #foundationmodels #recsys #machinelearning #research

157

1 Comment

We just released a blog on whether the use of multimodal feature transfer can detect deception (phenomena like sarcasm, irony, and condescension) in conversation. The short answer... Yes. This particular write-up profiles a subset of our work at Symbl.ai on multimodal large language models, and is research led by one of our summer interns, Benjamin R. (from Prof. Larry Heck's AVA Lab at Georgia Institute of Technology). We have a paper currently under submission on this work, and we can't wait to share the full details with all of you soon. Blog Link: https://lnkd.in/g6m_wFDi #multimodal #featuretransfer #LLMs #deception #conversation #sarcasm #irony #language #AI #ML #NLP #LargeLanguageModels #SummerInternship #ResearchPapers #ACL #Llama2 #SymblDotAI

16

Just Arrived in Vienna for #ICML2024, I will be presenting our work on improving GNNs' generalization ability with ensemble learning (https://lnkd.in/gimXe5fS) at Oral 5A (Thu 25 Jul 10:45 a.m. — 11 a.m.) and Poster 2407 (Thu 25 Jul 11:30 a.m. — 1 p.m.). Paper title: From Coarse to Fine: Enable Comprehensive Graph Self-supervised Learning with Multi-granular Semantic Ensemble. A heartfelt thank you to all my collaborators for their incredible support; this work would not have been possible without them. I am also looking forward to reconnecting with old friends and making new ones. If you are attending ICML or are interested in my research, please join me during my presentation session. Additionally, please feel free to reach out to exchange ideas if you are interested in topics such as (1) improving graph model generalization across different datasets, and (2) foundational models for time series forecasting. See you in Vienna! 🇦🇹 #ICML2024 #MachineLearning #ArtificialIntelligence

14

Paper of the week : Inference Scaling for Long-Context RAG - uses two strategies to investigate scaling laws for RAG: in-context learning (DRAG) and iterative prompting (IterRAG); finds that RAG performance consistently improves with the expansion of the effective context length under optimal configurations; when optimally allocated, increasing inference computation can lead to linear gains in long-context RAG performance; this leads to the development of a computation allocation model that can provide practical guidance for optimal computation allocation in long-context RAG scenarios. https://lnkd.in/dTfTRjp7 #paperoftheweek #machinelearning #llms

13

My latest research work, "Towards smaller, faster decoder-only transformers: Architectural variants and their implications" is now available on arXiv. In this work, we introduce architectural variations to the decoder-only transformer architecture for easier deployment, faster training and faster inferencing with little drop in performance. Do check it out in the link below and feel free to share your thoughts and insights. https://lnkd.in/gF_A4HeU #research #machinelearning #deeplearning

51

2 Comments

Reality Collapse 🌐 We're on the brink of a big change. Generative models will soon let people create their own personalized realities through text, video, and music. Generative models still suck at generating coherent long-form videos, but it's largely a matter of time. This week alone: 👉 Luma Labs introduced their 'Dream Machine', a decent 30-second video generator: https://lnkd.in/g9usVgPx 👉 Microsoft achieved human-level zero-shot voice cloning: https://lnkd.in/gj56K5Jz Very soon everyone can live in their own world. Society is shifting from a shared identity to individual, AI-driven experiences - what Jack Clark called reality collapse. Strange times ahead. #AI #GenerativeModels #RealityCollapse #FutureTech 🚀

3

This feature is very under-rated and I can’t emphasize how much this is a game changer. Impressive!. It’s very good at creating visualizations and small apps. This feature by itself puts claude one step ahead of ChatGPT regardless of the benchmark results of Sonnet over GPT-4o. #claude #LLM #ChatGPT

4