Mani Vannan

Semantic Search uses embeddings from transformer models and cosine similarity to capture the semantic meaning and relevance of the documents to the query

6

AI Agents aren't going to be easy, and the problem is the LLMs. LLMs are trained to provide an answer to a question or an output for a task. "write this code", "categorize this list". They are not, however, great at making decisions - navigating a fork in the road. In fact, they're trained to *not* make decisions for you. What makes an agent an agent is its ability to decide, and the current models are fairly terrible at it. There isn't as much training data on making decisions. The fine tuning isn't being done on decision making. For a while, agents are going to be rudimentary at best.

16

1 Comment

Unique session by Nicolas Boucher and Ilyas Iyoob, PhD It was not hype; it was actual lessons learned from implementing AI. --> For mission-critical systems with AI, there needs to be a barrier to ensure that human validates the AI repository --> When processing documents for scoring, avoid over-contextualization so the next scoring isn't biased from previous processing. A use case from Data Crafters!! --> For Forecasting, there are better options than Gen AI. Traditional time series and regression-type Forecasting are better approaches. --> You can get started even if you don't have the resources of big companies; you can get started with available tools. You can see more : https://lnkd.in/gt6tCEyv Note: Learning #python would come in handy. --> Starting with structured data would go a long way, so investing in #PowerBI will go a long way in feeding your data into your AI models. Get started today on your journey!!

14

4 Comments

🚨 LLM Deep Dive Series (2 of 7): The Architecture of LLMs: From Transformers to Massive Parameters 🧠🤖 In the second post of the LLM Deep Dive Series, we’re breaking down the architecture that powers Large Language Models (LLMs)—Transformers. Introduced in 2017 by Google's Research, the Transformer model has revolutionised natural language processing (NLP), serving as the foundation for advanced models like GPT-4. 🚀 What Makes Transformers So Powerful? The key innovation behind Transformers is the self-attention mechanism. Unlike previous models (like RNNs and LSTMs), which process input sequentially, Transformers handle entire sequences simultaneously, capturing relationships between tokens across long distances in text. ➡ Self-attention: allows the model to weigh the importance of different words in a sentence—so it knows which words to "focus" on to generate the most accurate predictions. This ability to evaluate all tokens at once enables Transformers to handle complex language tasks with remarkable efficiency. 🧠 Inside a Transformer Transformers rely on three key components: ➡ 1. Input Embeddings & Positional Encoding: Words are converted into vectors (embeddings), with positional encoding added to preserve word order. ➡ 2. Self-Attention: This mechanism helps the model "attend" to different words based on their relationships within the sentence. ➡ 3. Feed-Forward Networks: After the attention mechanism, each word goes through a fully connected network to further refine its meaning. These components are stacked into multiple layers, allowing Transformers to deepen their understanding of language with each pass. Scaling with Massive Parameters The true power of LLMs like GPT-4 and Sonnet comes from scaling. The more parameters a model has, the more information it can store and the better it performs. GPT-3, for example, has 175 billion parameters, allowing it to handle everything from generating human-like text to answering complex queries. However, as these models grow, so do the computational demands. More parameters require more computing power, energy, and data. As we scale, we're also seeing emergent behaviours—capabilities that aren’t explicitly programmed but arise naturally due to the vast amount of training data. And nobody really knows the limit, as some companies are betting billions on how far it could go ... 🔍 Why Transformers Outperform Traditional Models Transformers excel because they: - Process sequences in parallel, improving efficiency. - Capture long-range dependencies across large amounts of text. - Scale effectively, with more parameters leading to greater complexity and performance. P.S. Check out my book, for a deeper dive into building a successful AI "Grow Your Business with AI" https://bit.ly/4b31PEG 📚 Stay tuned for more insights on how LLMs truly work beneath the surface! #AI #LLM #DeepLearning #Transformers #MachineLearning #AIArchitecture

4

New research shows that a semantic layer can make GenAI answers 3X more accurate than direct SQL database queries (16% to 54%)¹. You may wonder: What the heck is a semantic layer, and how do they improve GenAI accuracy? My new post below explores those questions. And today's announcement of AtScale's next-generation architecture points toward the future of semantic data. I was thrilled to learn about it at my first meeting as a member of the AtScale board -- it's an amazing piece of engineering, including: 1️⃣ Generative AI and LLMs support acting as a metadata hub and language for enhanced reliability and accuracy.  2️⃣ A container-based architecture with new deployment options via Kubernetes or Docker and seamless integration with Snowflake, Databricks, and Google BigQuery.  3️⃣ Enterprise integration and authentication through open-source packaging for Keycloak, OpenTelemetry OpenAPI, and KeyGen.  4️⃣ dbt Labs Metrics Translator that allows dbt semantic models to integrate seamlessly with AtScale’s live query support for Tableau, Power BI, and Excel.  5️⃣ Semantic Integrated Development Environment (IDE) with no-code and code-first data modeling within the same user experience, YAML-based modeling, and Git for full CI/CD support. Congratulations to Christopher Lynch and the entire AtScale team on this huge milestone! #semanticlayer #datasemantic #knowledgegraph CC: Christopher Lynch, Nicole Francoeur, Cort Johnson, Pete Perrone, Eagen, Michael, Ryan Floyd, Roseann Palmieri, David P. Mariani

55

16 Comments

Accelerating LLMs by 2x with Graph-structured Speculative Decoding. Researchers have found a way to make speculative decoding up to 2x faster by generating multiple hypotheses and merging them into a directed acyclic graph (DAG).

8

One of the most promising directions in software engineering is merging stateful architectures with LLMs to handle complex, multi-step workflows. While LLMs excel at one-step answers, they struggle with multi-hop questions requiring sequential logic and memory. Recent advancements, like O1 Preview’s “chain-of-thought” reasoning, offer a structured approach to multi-step processes, reducing hallucination risks—yet scalability challenges persist. Configuring FSMs (finite state machines) to manage unique workflows remains labor-intensive, limiting scalability. Recent studies address this from various technical approaches: 𝟏. 𝐒𝐭𝐚𝐭𝐞𝐅𝐥𝐨𝐰: This framework organizes multi-step tasks by defining each stage of a process as an FSM state, transitioning based on logical rules or model-driven decisions. For instance, in SQL-based benchmarks, StateFlow drives a linear progression through query parsing, optimization, and validation states. This configuration achieved success rates up to 28% higher on benchmarks like InterCode SQL and task-based datasets. Additionally, StateFlow’s structure delivered substantial cost savings—lowering computation by 5x in SQL tasks and 3x in ALFWorld task workflows—by reducing unnecessary iterations within states. 𝟐. 𝐆𝐮𝐢𝐝𝐞𝐝 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧 𝐅𝐫𝐚𝐦𝐞𝐰𝐨𝐫𝐤𝐬: This method constrains LLM output using regular expressions and context-free grammars (CFGs), enabling strict adherence to syntax rules with minimal overhead. By creating a token-level index for constrained vocabulary, the framework brings token selection to O(1) complexity, allowing rapid selection of context-appropriate outputs while maintaining structural accuracy. For outputs requiring precision, like Python code or JSON, the framework demonstrated a high retention of syntax accuracy without a drop in response speed. 𝟑. 𝐋𝐋𝐌-𝐒𝐀𝐏 (𝐒𝐢𝐭𝐮𝐚𝐭𝐢𝐨𝐧𝐚𝐥 𝐀𝐰𝐚𝐫𝐞𝐧𝐞𝐬𝐬-𝐁𝐚𝐬𝐞𝐝 𝐏𝐥𝐚𝐧𝐧𝐢𝐧𝐠): This framework combines two LLM agents—LLMgen for FSM generation and LLMeval for iterative evaluation—to refine complex, safety-critical planning tasks. Each plan iteration incorporates feedback on situational awareness, allowing LLM-SAP to anticipate possible hazards and adjust plans accordingly. Tested across 24 hazardous scenarios (e.g., child safety scenarios around household hazards), LLM-SAP achieved an RBS score of 1.21, a notable improvement in handling real-world complexities where safety nuances and interaction dynamics are key. These studies mark progress, but gaps remain. Manual FSM configurations limit scalability, and real-time performance can lag in high-variance environments. LLM-SAP’s multi-agent cycles demand significant resources, limiting rapid adjustments. Yet, the research focus on multi-step reasoning and context responsiveness provides a foundation for scalable LLM-driven architectures—if configuration and resource challenges are resolved.

168

7 Comments

[arXiv] RAGChecker is a fine-grained evaluation framework for diagnosing retrieval and generation components within Retrieval-Augmented Generation (RAG) systems. The results demonstrate that RAGChecker strongly correlates with human judgment and provides valuable insights into the design trade-offs of different RAG architectures. https://lnkd.in/gPCx4wBS

10

Wondering how to improve exponentially the performance of your LLMs???? Your solution could be CRITIC! 🤖🗣️🔧 CRITIC allows LLMs, which are essentially “black boxes” to validate and progressively amend their own outputs in a manner similar to human interaction with tools. More specifically, starting with an initial output, CRITIC interacts with appropriate tools to evaluate certain aspects of the text, and then revises the output based on the feedback obtained during this validation process. This offers an element of getting external feedback; in many cases this gives much better results than only using LLMs to critique itself. You should take a look to this cheatsheet by Val Andrei Fajardo, ML Engineer at LlamaIndex, that guides you through all the most relevant components of CRITIC. 📝 You can take a look to the paper below 👉 https://lnkd.in/dhskQ8rw #ai #genai

57

6 Comments

🚀 Transforming Healthcare with AI: Precision and Progress 🚀 * Rapid Drug Development: AI reduced the drug discovery process to just 30 months with Insilico Medicine, compared to the traditional decade-long timeline. *Enhanced Diagnostic Accuracy: In diagnosing complex cases, AI accurately identified tricky diagnoses 39% of the time and provided the correct diagnosis as a secondary option 64% of the time, surpassing traditional methods. *Focused Patient Care: Ambient AI allows doctors to focus more on patients by handling administrative tasks, shown in a study where 3,442 physicians adopted this technology for over 303,266 visits. *Empathetic Patient Communication: AI proved to make patients feel more heard, offering emotional support effectively, though it showed a reduction in perceived empathy when users knew the communication was AI-generated. #AIinHealthcare #HealthTech #PatientCare #Innovation Summary from: https://lnkd.in/gyqbBf5R

20

The 3-Step Guide to Making Your Company AI-Forward 🚀: 1. Provide Access to ChatGPT Plus / Claude Pro($20/user/month) 2. Empower People to Experiment 💡 3. Encourage Sharing 📢 Sure, it sounds simple—maybe too simple—but many executives are overthinking it, bogged down by "CEO guides." Every day, people are discovering nuanced AI workflows that collectively save weeks of time across the company. This isn't a far-off dream; it’s happening now. Why wait until 2040? The AI tools you need are available today. And guess what? Early adopters are eager to experiment. The excitement around AI tools is still very much alive. Let your team experiment. Give them a platform to share their insights. It's that simple! Transform your business with AI today! 🚀 #ArtificialIntelligence #Startups #Technology #AIFuture #Innovation

5

1 Comment

The Artificial Intelligence Index Report 2024 by Stanford University is out For those who like charts and numbers, 502 pages of joy. Link to 40 MB PDF: https://lnkd.in/gChEvk9Z Stanford Institute for Human-Centered Artificial Intelligence (HAI) #artificialIntelligence #AI #spe #petroleumEngineering #machineLearning

10

AI at Meta has released a white paper on the Llama3 405B model, and it provides a lot of granularities on the compute infrastructure they used for training. It took 54 days on a cluster of 16,000 H100 GPUs to train this 405 billion parameter model, the largest open model available currently. 🔹 Compute: Meta used a cluster of 16K NVIDIA H100 GPUs, each running at 700W TDP with 80GB of HBM3, on Meta's Grand Teton AI server platform. Each server has 2 CPUs and 8 GPUs and the GPUs are connected via NVLink. 🔹 Storage: Tectonic, Meta’s general-purpose distributed file system is employed for pre-training, offering 240 PB of storage across 7,500 servers with SSDs. The storage fabric supports a peak throughput of 7 TB/s and a sustained throughput of 2 TB/s. The biggest challenge in the AI storage fabric is supporting the highly bursty checkpoint writes that saturate the storage fabric for short durations. Checkpointing saves each GPU’s model state, ranging from 1 MB to 4 GB per GPU, for recovery and debugging. 🔹 Network: Llama 3 training cluster uses Ethernet as the server interconnect! Yes, RDMA over Converged Ethernet (RoCE) fabric based on the Arista Networks 7800 (Broadcom Jericho) and Minipack2 OCP switches. Smaller models in the Llama family leveraged the Nvidia Quantum2 InfiniBand fabric. Both Ethernet and InfiniBand clusters operate at 400 Gbps, and Meta has done a lot of optimizations on the #RoCE fabric to make it work efficiently on large GPU clusters. 🔹 Llama3 405B Inference: When using a BF16 number representation for the model parameters, Llama3 405B does not fit in the GPU memory of a single machine with 8 Nvidia H100 GPUs. To address this issue, Meta parallelizes model inference using BF16 precision across 16 GPUs on two machines. Within each machine, the high NVLink bandwidth enables the use of tensor parallelism. Across nodes, however, connectivity has lower bandwidth and higher latency, so they use pipeline parallelism instead. Now the most interesting part! There were 419 interruptions during this training cycle. And about 78% of unexpected interruptions were attributed to confirmed or suspected hardware issues! There were 256 GPU related failures! NVIDIA’s GPUs are highly optimized for large clusters and for AI training jobs. No wonder other GPU vendors struggle to get into the AI training space. It is indeed hard to make a large cluster work seamlessly. The root-cause categorization of unexpected interruptions during the 54-day period of Llama3 405B pre-training is in the table below. Here is the link to the full white paper: https://lnkd.in/gJN_2GgT

147

5 Comments

Reflection agents in LangChain are designed to improve the quality and success rate of AI agents by incorporating a process of self-evaluation and critique. Here's how they work: **-Key Concepts-** 1. Prompting Strategy: Reflection agents use a prompting strategy where an LLM (Language Model) is prompted to reflect on and critique its past actions. This can involve reviewing previous outputs and incorporating external information, such as tool observations. 2. System 1 and System 2 Thinking: The concept is inspired by the idea of "System 1" (reactive, instinctive thinking) and "System 2" (methodical, reflective thinking). Reflection agents aim to move closer to System 2-like behavior by encouraging more thoughtful and deliberate responses. 3. Feedback Loop: Reflection agents create a feedback loop where the agent evaluates its actions, receives feedback, and uses this information to refine future actions. -**Implementation in LangGraph-** 1. Simple Reflection: This involves composing two LLM calls: a generator and a reflector. The generator responds to the user's request, and the reflector, acting as a teacher, offers constructive criticism. This loop can run multiple times to refine the output. 2. Reflexion Architecture: This architecture, developed by Shinn et al., allows the agent to learn through verbal feedback and self-reflection. The actor agent critiques each response and grounds its criticism in external data. 3. Language Agent Tree Search: This technique involves searching through possible actions and responses to find the most effective solution, incorporating reflection at each step. -**Benefits-** 1. Improved Output Quality: By reflecting on past actions, agents can produce higher quality and more accurate outputs. 2. Enhanced Decision-Making: Reflection helps agents make more informed and deliberate decisions, improving overall performance. 3. Adaptability: Agents can adapt to new information and feedback, making them more flexible and effective in various scenarios. Reflection agents are particularly useful for knowledge-intensive tasks where response quality is more important than speed. They help break out of purely reactive patterns and encourage more thoughtful and methodical responses. #agenticapp #ReflectionAgents #langgraph #langchain #flowengineering #llm #genai

12

3 Comments

AIOS - Architecture of Interoperable Information Systems (now available for free download)   📚 In 2010, I submitted my doctoral thesis, introducing the Architecture of Interoperable Information Systems (AIOS), a reference architecture for digitalizing large interoperable enterprise systems. Topic: If enterprises want to engage in cross-organizational business process automation, e.g. in B2B-scenarios, the involved digital systems must be able to work together, i.e. they need to be interoperable. The question is, how to achieve this? How does a sustainable, efficient, and secure design of these complex digital ecosystems look like? And, for creating this landscape, which layers must be addressed, in which sequence?    💻 These were the heydays of #ModelDrivenArchitcture, #ServiceOrientedArchitecture (SOA) and the #WebServiceStack. The latter contained sophisticated XML-based standards, describing for example service contracts (WSDL), business objects (XSD), business processes (BPEL Frank Leymann Jan Mendling), and roles and rights (XACML). Here also elegant concepts were developed for decoupling and correlating internal process elements ("private processes") and their adjacent public process elements ("public processes") visible in a collaboration, alongside a comprehensive view that integrates complementary public processes ("global processes"). 🏛 Complementary, the "Architecture of Integrated Information Systems" (ARIS), introduced by August-Wilhelm Scheer outlined the foundational pillars for the architecture of large digital enterprise systems from a business perspective: Business functions, business objects ("data view"), business processes, and roles and rights ("organization view"). According to ARIS (and similar to #MDA), these views needed to be addressed on three levels: business design, technical design, and implementation. 🚀 AIOS integrates the core concepts from Service-oriented Architecture, MDA, Collaborative Business, and ARIS. It serves as a generic blueprint for large organizations to develop interoperable information systems by systematically adjusting and extending their internal information systems. AIOS is product- and vendor-independent, providing a generic framework for the layers, views, relationships, and technical means necessary to establish interoperable information systems efficiently. 🙏 AIOS drew on the findings of various research projects from the #EuropeanUnion on #interoperability, e.g. #ATHENA, #R4eGov, and #Interop. Fun days with great colleagues, e.g. Christine Legner Kristin Weber Ulrike G. Thomas Knothe Maria Wimmer. Now, the Logos publishing house has generously made AIOS available for free download; please find the link in the comments section below.   #Interoperability #EnterpriseArchitecture #enterpriseDigitalization #processAutomation #dataContracts #EAMfundamentals #AIOS #ARIS

39

2 Comments

Gemini 1.5 Flash’s 1 mllion token context window, low latency, and cost efficiency have quickly made it a favorite among our customers. Check out the advantages of Gemini 1.5 Flash over comparable models like GPT 3.5 Turbo: ⚡ Gemini 1.5 Flash's 1 million token context window is approximately 60x bigger than the context window provided by GPT-3.5 Turbo ⚡ On average, Gemini 1.5 Flash is 40% faster than GPT-3.5 Turbo when given input of 10,000 characters ⚡ Gemini 1.5 Flash is up to 4X lower on input price than GPT-3.5 Turbo, with context caching enabled for inputs larger than 32,000 characters. Learn more about Gemini 1.5 Flash—now generally available on Vertex AI ↓

1

2 Comments

What is best data labeling and what is it used for? What exactly does data labeling mean? I... https://lnkd.in/gSraXiUt #24x7offshoring #ai #Data #labeling What exactly does data labeling mean? Is it really just labeling data, like it literally says? Well, it’s kind of a yes and no. What is Data Labeling? Essentially, data annotation is the process of labeling material content that can be identified through computer vision or natural language processing (NLP). When we label or annotate these types ... Read more

4