Ching-Wei Chen

💡 Enhancing Retrieval Accuracy with Contextual RAG As GenAI systems go beyond demos and are deployed at production scale, improving retrieval accuracy remains a top priority for building robust Retrieval-Augmented Generation (RAG) systems. One promising technique that Anthropic recently discussed on their blog? Contextual RAG. The idea is to run LLMs on document chunks first to create context before running them through the standard embeddings pipeline. This method alone can reduce failure rates by 35%. And adding additional techniques like BM25 indexing and Reranking - techniques that are already well researched - improves failure rates by 67% 🤯 As token costs continue to drop and smaller models improve, these kinds of approaches become increasingly viable for large-scale use cases. You could, for example, use a smaller LLM for context creation while leveraging a larger, more powerful model for generating responses. Combining models in this way can help strike a balance between performance and cost. #AI #LLMs #RAG #ContextualAI #ScalableAI

24

3 Comments

Attention - I mean RNNs are all you need??!! Are new linear RNN based models going take over the top spot for the model of choice in the future? Specifically Mamba? Dynamic 'Linear' RNN's based models with compute of O(n log(n)) instead of O(n^2) (transformers) are starting to take off but are meeting some resistance in the peer review process. How did they overcome the two main issues with RNN's- lack of parallel computations and the training difficulty arising from only learning from recent inputs, making long sequence problems intractable and reserved for transformer and cnn based models)? Read about how they work in the papers below- Linear RNN's(state space models) : https://lnkd.in/gechNx9V Mamba LLM based on dynamic linear RNN's (no attention, no MLP needed): https://lnkd.in/g8dSmhZt

4

$2 H100s: How the GPU Bubble Burst Last year, H100s were $8/hr if you could get them. Today, there's 7 different resale markets selling them under $2. What happened? The market has flipped from shortage ($8/hr) to oversupplied ($2/hr), because of reserved compute resales, open model finetuning, and decline in new foundation model co’s. Rent instead. https://lnkd.in/gpQEidmC

2

1 Comment

1. IMO, agent patterns are one of the most interesting ways to extend LLMs. memary is based on leveraging a graph database (neo4j) for long-term memory (to get around the limited memory inherent in even the largest context windows in LLMs) and augmenting it with other tools like web search and Computer Vision. https://lnkd.in/gfFcBqJx Looks interesting. 2. More about how consolidation effects the entire healthcare industry: https://lnkd.in/gXJKqNJB The Change hack/ransomware attack not only affected claims, service, refills, etc. but now CMS & NCQA are delaying the entire HEDIS reporting process (used for Medicare/Medicaid quality ratings) due to the impact. 3. NARA bans ChatGPT use agency-wide. https://lnkd.in/gW8T94JU Federal Government agencies see this as a risk, including their information leaking and being used to train models (which may then output the wrong information or show PII/PHI other proprietary information). 4. Oh great, now I can't even be angry anymore :) https://lnkd.in/gcrN9mjx

1

1 Comment

How do you make large-scale knowledge graphs actually usable for AI retrieval? We’ve been tackling this question with memory fragment projection using cognee. The idea is simple: 💡 Instead of analyzing the entire graph, we extract focused subgraphs (memory fragments) that are small enough for in-memory exploration but rich enough to support AI queries. In the blog, Hajdu László, Ph.D. walks us through: 🚶‍➡️ Subgraph filtering methods (community-based, neighborhood-based, etc.) and their tradeoffs 🚶‍♀️‍➡️ How cognee integrates with Neo4j to detect communities and project graph fragments 🚶‍♂️‍➡️ A practical example of projecting memory fragments from Neo4j’s knowledge graph This approach lets us: ✅ Optimize AI retrieval pipelines with GraphRAG ✅ Maintain flexibility while focusing on relevant graph layers ✅ Scale down the complexity without losing context If you’ve worked with knowledge graphs or retrieval pipelines, you know the pain of balancing scale and precision. This solution makes that process much more approachable. Here’s the blog 👇 https://lnkd.in/d5js2psf Would love to hear your feedback or challenges you’ve faced in this space!

21

This new little product demo is awesome! Our new Rev AI Transcript Assistant will save you TONS of time. Easily pull out nuggets of info from your transcripts, making valuable content more accessible than ever. https://lnkd.in/gzzFpcDa #transcription #asr #accessibility #transcript #summarization #ai #assistant

31

2 Comments

🤔 Ready to build smarter, more efficient agents? My latest blog post dives into the world of smaller, task-specific LLMs designed to power the next generation of agent-based systems. 💪 Discover how these models enhance agent capabilities while minimizing resource consumption. #AI #LLM #AgentBasedSystems #DeepMind

38

3 Comments

🌶️ Hot take: Claude's Projects feature is the sleeper hit of 2024 (and way more useful than Artifacts which gets all the hype)! Not only that, the Projects feature is laying the groundwork for where generative AI is headed, solidified by Anthropic's launch of Model Context Protocol (MCP). Not sure what I'm talking about? I've got you - check out my thoughts on Projects, the feature and the future we're not talking about enough: https://lnkd.in/d9knnf-U

8

It’s finally here! I’m thrilled to introduce you to Llama 3, our latest generation of models that is setting a new standard for state-of-the art performance and efficiency for openly available LLMs. Key highlights: • 8B and 70B parameter openly available pre-trained and fine-tuned models • Trained on more than 15T tokens, 7x+ larger than Llama 2's dataset! • State-of-the-art performance across industry benchmarks • New capabilities, including enhanced reasoning and coding • 3x more efficient training than Llama 2 • New trust and safety tools with Llama Guard 2, Code Shield, and CyberSec Eval 2. • Training underway on a leading-edge 400B+ parameter model  We’re also releasing a new version of Meta AI powered by Llama 3. We’re making it available in 13 new countries, integrating enhanced search capabilities and adding new ways to experience it on the web at meta.ai. Our latest updates to image generation on Meta AI is state of the art, allowing you to create, animate and share stunning images with a simple text prompt in seconds. We think Meta AI is the world’s most intelligent AI assistant and we can't wait for you to try it and tell us what you think. Read more about it here: https://lnkd.in/eB5yzi8R

2,385

80 Comments

🚀 Transform AI Conversations: Join Our Exclusive vCon Use Cases Zoominar on May 14th! 🗓️ Attention trailblazers in the world of AI and communication! 🌟 Following the resounding success of our "Introduction to vCon" Zoominar, Thomas McCarthy-Howe and I are thrilled to invite you to Part II of our groundbreaking series: "vCon Use Cases: Revolutionizing AI Conversations". 🎉 As a pending Internet Engineering Task Force (IETF) RFC, virtualized conversations (vCons) are set to become a critical building block for AI task managers, revolutionizing the way we store, analyze, and leverage customer dialogues in AI applications. 📈 Join us on May 14th at 1 PM Eastern US for an interactive online Zoominar, where we'll explore the transformative use cases of vCons across various domains: 🌟 Customer Experience: Conversational CRM Listening to Customers Mass Customization 🌟 Efficiency: Compliance and Consent Generative AI 🌟 Observability and Transparency: Robot Food Enterprise Search Agent Governance 🌟 Business Transformation: New Revenue Business Improvement Throughout the Zoominar, you'll have ample opportunities to ask questions, share your insights, and engage in thought-provoking discussions with our expert speakers and fellow attendees. 💬 🔥 Limited spots available! Secure your place now by registering here: https://lnkd.in/ec5q5ErV 🔥 If you missed our previous Zoominar, "Introduction to vCon," don't worry! Simply send an email to [email protected], and we'll provide you with exclusive access to the recording. 📩 Seize this opportunity to explore the transformative potential of vCons and stay at the forefront of AI and communication innovation. Let's unlock new possibilities together! 🔓 #vCon #ArtificialIntelligence #CX

22

Very useful step by step tutorial if you plan to finetune LLM on your data to improve model quality

19

NVIDIA's very own proprietary LLM is here. But how does NVLM 1.0 stack up against competitors like LLaMA? 👁️‍🗨️ NVLM 1.0 is probably still best suited for an enterprise setting. It excels in multimodal tasks, delivering top performance on complex datasets and achieving rapid inference speeds with NVIDIA's H100 GPUs. This makes it ideal for high-throughput, real-time applications. LLaMA 3.2, on the other hand, is all about efficiency and flexibility. It shines in mobile and edge environments with a lightweight architecture that balances power consumption and speed. That means it's perfect for developers working on constrained devices or looking for open-source flexibility. So, at the end of the day… which is best for you? It comes down to whether you need enterprise-level performance or scalable, low-power efficiency. Have you gotten your hands on either of these models? I'm curious to hear what you think. #AI #NVIDIA #Meta #GenerativeAI #MachineLearning

19

Read this blog post by Ercüment İlhan to learn how a 70 year old algorithm helped make our contextual bandit powered personalization service ~40 times faster. https://lnkd.in/eGps5AJt

43

1 Comment

𝙒𝙝𝙚𝙧𝙚'𝙨 𝙩𝙝𝙚 𝘼𝙄? isn't always the right question. Sometimes it's better to ask 𝙒𝙝𝙮 𝙩𝙝𝙚 𝘼𝙄? Rabbit R1, and Humane AI Pin just before it, are a great example of that. Problems like ordering on DoorDash can reliably be solved with widely adopted and high performing APIs. It's almost as good as the existing tech but with AI is not a solution anybody wants or needs. https://lnkd.in/gZnveh6F I was reminded of this when working on an #llm project for one of our partners. Holding an #ai hammer it's easy to see everything as a nail. But a big chunk of the problem was actually better suited for traditional data processing methods, with #genai only applied to specific challenges that were hard to solve with traditional engineering. The best advice for every project is to use the right tech instead of getting overexcited for the latest tech. This very much applies to AI. This simple approach can lead to solutions that are more efficient, salable and cost efficient.

5

If you think compute scaling will plateau, think again. We are in the baby states of inference time compute. VLLMs coupled with computer tools, reward learning and RL can leverage orders of magnitude more inference time compute than what we have seen with foundation models. Here’s a concrete training architecture (replace \pi with a large pre-trained vLLM and add it to the reward learner too, \R is the space of all computer tools)

22

1 Comment

This is a great first step in understanding how scaling dynamics extend beyond language to other types of sequential data. Time is a universal axis, understanding how models learn optimally over data collected over time is a crucial step in the broader AI research agenda.

26

Excited to announce another major upgrade to WhyHow.AI’s Knowledge Graph SDK - tying vector chunks to graph nodes automatically, for a more deterministic and richer context window. Check out how we do it, why we did it, and an example benchmark of the increased completeness of the answer. Tired of just returning single-word triples from your knowledge graph? WhyHow.AI’s latest upgrade with vector chunk linking now lets you use a graph structure to determine which raw vector chunks to return to the context window, combining the best of knowledge graphs and vector search. “While the triples in a Knowledge Graph are useful in providing specific information that semantic similarity was unable to retrieve, we wanted to also allow leeway in the information represented and retrieved from the graph, to include the surrounding words and retrieving the relevant raw vector chunk tied to that graph node as well. By tying vector chunks to a knowledge graph, we get the advantages that lie in both vector and graph search.” - WhyHow.AI Design Partner WhyHow.AI builds workflow tools for data orchestration, and graph creation, and we work on top of any data extraction model you want to bring. In this case, we work on top of OpenAI, Neo4j, and Pinecone, and will be supporting the most popular data extraction models, LLMs, graph and vector databases. https://lnkd.in/eEJdUNPi

69

6 Comments