David Feldman

Got a chance to write for InfoWorld on why RAG - enabled AI agents are forming the foundation of successful enterprise genAI deployments to automate end to end knowledge workflows. “...the real magic of RAG is in the retrieval model and its upstream components. RAG deployments live and die by the quality of the source content and the retrieval model’s ability to filter the large data source down to useful data points before feeding it to an LLM…” “...a well-designed AI agents approach to the automation of complex knowledge workflows can help mitigate risks with RAG deployments by breaking down large use cases into discrete “jobs to be done…” Auquan is helping teams across private markets, investment banks, insurance firms and other financial services extract meaningful insights from large complex, noisy data quickly and cost effectively, by leveraging RAG and AI agents. If you are similarly looking to buy or build infra to transform troves of valuable unstructured data into timely, actionable intelligence for your firm, give us a shout. Article: https://lnkd.in/gfsis8QB #EnterpriseAI #RAG #AIagents #GenAI #finance

39

I highlight the surprising effectiveness of simple chunking strategies over more complex semantic methods for fine-tuning and RAG systems. Original Problem: Fine-tuning & RAG systems often rely on semantic chunking, which splits documents based on topic coherence, with the goal of enhancing retrieval quality. However, this approach's computational cost is high, and there’s little evidence proving it consistently outperforms straightforward, fixed-size chunking. Overview: Three chunking strategies—fixed-size, breakpoint-based semantic, and clustering-based semantic—across document retrieval, evidence retrieval, and answer generation tasks on multiple datasets. Key Insights: Simplicity Wins: Fixed-size chunking frequently outperformed semantic chunking, particularly on real-world documents. Context Dependency: Semantic chunking’s effectiveness varies greatly with context and isn’t universally beneficial. Quality over Method: Embedding model quality affects performance more than the chunking strategy itself. Efficiency Matters: The computational burden of semantic chunking often outweighs its minor benefits. Results: Fixed-size chunking excelled in evidence retrieval tasks, outperforming on 3 out of 5 datasets. Semantic methods only showed slight advantages on artificially constructed datasets, and performance differences across strategies remained marginal. In summary, for most practical applications, fixed-size chunking provides a more efficient and equally effective alternative to complex semantic chunking. https://lnkd.in/e-UBHx8W

48

3 Comments

LLMs are trained to do complex things and tokenize language strings for maximum efficiency. This is important to understand when you are writing prompts. This example with Gemini will show you what I mean. Keep this in mind if you are struggling with detecting hallucinations in your work.

2

1 Comment

Crypto 101 - Mapping Groups: Adding, Subtracting, Multiplying, Dividing and Exponentiation: https://lnkd.in/e3sSYqpp

2

This is an excellent talk and demo from Emil Eifrem, Neo4j CEO, on why GraphRAG is an important component in grounding an Enterprise GenAI LLM. #neo4j #genai #GraphRAG

3

1 Comment

Interesting piece on how vector databases (an esoteric technology) have finally found use in the generative AI era ---- Vector databases have been around for decades, but are now emerging as something of an industry standard for AI businesses to use alongside a technique called retrieval-augmented generation, or RAG. When combined, businesses can link their private data with large-language models like OpenAI’s GPT-4, allowing the AI to perform data analysis, summarization and other tasks on their data. Without them, AI models are limited to what they have learned from their initial training on public data online, up to a certain point in time, and are more prone to factual errors called “hallucinations.” Vector databases are different from traditional databases with columns and rows because they are designed to store a massive amount of data as “vectors,” or numerical representations of the raw data New York-based startup Pinecone was an early entrant in the vector database AI space. Valued at $750 million after raising $100 million from Andreessen Horowitz and other investors last April, Pinecone has been building its vector database since 2019—a fortunate few years before OpenAI’s ChatGPT kicked off the AI gold rush in late 2022. ----- This one of those things you may learn about in computer science, along with red-black trees, and never ever use! #ai #chatgpt #llms #databases

8

3 Comments

I've posted on the work that Anthropic is doing to improve LLM interpretability, and again want to stress how important this is to the #GenAI field, especially when applied in regulated industries. The notorious "black boxes" of neural networks make them nearly impossible to understand why they generate what they do. (Note: even Sam Altman does not understand why the GPT models work so effectively. For that matter, no one actually does!!) Here is a pretty technical explanation of the approach that Anthropic is taking. I hope that more researchers dig into this. BTW - during my graduate work in genomics and drug development I discovered that a reason we don't have more/better drugs for genetic diseases is that most of them are not monogenic, meaning a disease is typically caused by multiple, even many, genes. What Anthropic is discovering seems to also be the case in LLM's. i.e., the outputs generated arise from a very complex interplay of many 'genes' in the model.

5

A few days ago Microsoft released the first version of BitNet.cpp which is a framework for running AI models on CPU. It currently depends on models being quantized down to 1-bit precision, but the potential is huge as this makes it possible to run (eventually) sophisticated models on not just consumer hardware, but old consumer hardware without the latest GPU tech. Think about everyone without an iPhone 15 pro+ being able to run Apple Intelligence 🍏 😲 Microsoft obviously gets a lot of attention because of its partnership with OpenAI, but they're also quietly leading innovation in the resource-constrained AI space - first releasing LoRA/PEFT which makes it possible to train/tune large models without massive amounts of GPU compute, to now BitNet which removes the need for GPUs altogether, very cool to see. Anyway I ran a quick inference test on a Macbook M3 running a Llama3 model using BitNet (top terminal/left CPU cores 7-11) vs using Ollama (bottom terminal/right GPU resources), and as it says on the tin 🥫 - BitNet only uses the CPU 🤷‍♂️ For those interested, BitNet.cpp repo here: https://lnkd.in/gUQWHnJY [Disclaimer: video is at 2x speed, and the 1bit Llama model gets into a bit of a repetitive loop 🔂 and then starts talking about cross-platform development ... but let's focus on the hardware breakthrough here, not the model 😂] #AI #MachineLearning #TechInnovation #BitNet #CPU #AIforEveryone #OpenAI #Microsoft

45

1 Comment

💥 New day, new model 💥 We ran our performance benchmarks on Anthropic Claude 3.5 Sonnet v/s OpenAI's GPT-4o & the results weren't what you'd expect ⏬ Anthropic says Claude 3.5 Sonnet outperforms all competitor models on key evaluations, at twice the speed of Claude 3 Opus and one-fifth the cost. In our benchmarks 3.5 Sonnet & 4o came out neck & neck. Ultimately, you need to run evaluations on your task at hand and measure the quality yourself. 🔍 Data Extraction: Both models identified 60-80% of data correctly, but neither excelled in this task. 📊 Classification: Claude 3.5 Sonnet (72%) outperformed GPT-4o (65%) in mean accuracy. However, GPT-4o led in precision (86.21%), crucial for accurately classifying customer tickets. 🧠 Verbal Reasoning: GPT-4o outperformed Claude 3.5 Sonnet with 69% accuracy, versus 44%. Both models did well on analogy and relationship questions. 📜 Context window: Claude 3.5 Sonnet boasts a larger context window (200K) compared to GPT-4o's 128K 🌪 Latency: Claude 3.5 Sonnet is 2x faster than Claude 3 Opus, but it’s still lags behind GPT-4o when it comes to latency (50% faster Time to First Token) Price and throughput for these models was very similar Want to learn more about how these models stack up? Check out our full analysis on the Vellum blog: https://lnkd.in/dwEcQJe4 Oh and did I say, Claude 3.5 Sonnet is already in Vellum? Sign up for a demo to run your evals & start building!

68

1 Comment

⁉️Generative coding LLMs are all the rage...but what security flaws could they be harboring ⁉️ While I utilize these tools just as much as any modern developer, I always double-check the output, once visually, and once via additional LLM prompts. ❓️Why is this important❓️ Take, for example, the potential risk when an AI coding agent hallucinates a Python library that doesn't exist, or worse, hallucinates a slightly different name for that library that is associated with a malicious copy-cat library...then automatically installs it in your project. This has already happened, but thankfully with no major impacts. From InfoWorld: "Hallucination could supercharge this, the researchers argue. Earlier in 2024, researcher Bar Lanyado of Lasso Security sent a small shudder through parts of the developer world when he discovered that several large companies, including ecommerce giant Alibaba, were using or recommending a Python software package called 'huggingface-cli'. The package was completely hallucinated. When he uploaded an empty package with the same name to a repository to test its popularity, it was downloaded more than 30,000 times in a three-month period. In other words, large numbers of developers were downloading an imaginary software package because an LLM had at some point hallucinated its existence to solve a specific programming task." ▶️Keep Calm and Always Review Your Code!◀️ #coding #agenticai #llm #hallucination

6

🎉 KAN: Kolmogorov-Arnold Networks ⚡️TL;DR: While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. 👉 Paper Discussion by First Author, Ziming Liu : https://lnkd.in/gYwKMHfw pip install pykan 👉 Abstract: Inspired by the Kolmogorov-Arnold representation theorem, we propose Kolmogorov-Arnold Networks (KANs) as promising alternatives to Multi-Layer Perceptrons (MLPs). While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. We show that this seemingly simple change makes KANs outperform MLPs in terms of accuracy and interpretability. For accuracy, much smaller KANs can achieve comparable or better accuracy than much larger MLPs in data fitting and PDE solving. Theoretically and empirically, KANs possess faster neural scaling laws than MLPs. For interpretability, KANs can be intuitively visualized and can easily interact with human users. Through two examples in mathematics and physics, KANs are shown to be useful collaborators helping scientists (re)discover mathematical and physical laws. In summary, KANs are promising alternatives for MLPs, opening opportunities for further improving today's deep learning models which rely heavily on MLPs. ⚙️ Code: https://lnkd.in/ghpeJjyU 🦋 Docs: https://lnkd.in/gTBdBfYh 📄 Paper: https://lnkd.in/gcwniZuS 💡 Note: Checkout the “Author’s Note” section on GitHub ReadMe. https://lnkd.in/g7F3HPr7 Also see: Althernate implementation of KAN on GitHub 👉 efficient-kan: https://lnkd.in/gjGEEkpX 👉 fourier-kan: https://lnkd.in/gbjh5x5i #KAN #paper #code #docs #python #research #physics #pinn #ml

1

In most embedding models, a single vector is generated for each document chunk, effectively capturing the chunk's overall meaning. However, this approach can miss out on fine-grained details, especially when critical contextual information is dispersed across the document. Similar to ColBERT's late interaction, which preserves token-level context, "Late Chunking" takes it a step further by pooling token embeddings that are fully context-aware of the entire document. This method significantly enhances retrieval performance by maintaining richer context within each chunk, leading to more accurate and meaningful results. #largelanguagemodel #naturallanguageprocessing #llm #nlp

8

This story can be used also as example for technical debt in software projects. I can’t say anything about the motivation of the author but, for sure I can state that any work around has a price! I will use Maxime Labonne post as example when talking about companies that try so sell you services without any sort of detail

6

Yann LeCun on Building AI Models with Common Sense Machine learning pioneer and chief artificial intelligence officer at Meta* Yann LeCun has emerged as the industry's most authoritative skeptic during the AI ​​boom. In a new interview (https://lnkd.in/drKvHDA6), he explains why existing AI models, while useful, won't be able to compete with human intelligence for a long time. 🐈 A cat is smarter than a computer Lecun doesn't take the threat to humanity from AI seriously. Even cats have an internal model of the physical world, persistent memory, and some ability to reason and plan — none of which are present in the most advanced AI models. In the future, gadgets and AI assistants should acquire a semblance of human intelligence and behave like living assistants, but it will take decades to create such systems, the scientist insists. 🛠 It's not about quantity The problem with today's AI systems is not their scale, but their architecture. No matter how many thousands of GPUs train existing models, they will never develop into strong artificial intelligence. Priority should be given not to large language models (LLM), but to other approaches. For example, at the Meta* Laboratory of Fundamental Research in AI, they are trying to train models on visual data so that artificial intelligence forms its own picture of the world, as happens with baby animals. 💬 Are LLMs so important? Large language models, such as ChatGPT and others, will play only a small role in the development of human-like systems with common sense. "We're used to the idea that those who can express themselves or have a command of language are intelligent, but that's not true. You can manipulate language and not be intelligent, and that's what LLMs demonstrate," says Yann LeCun.

1

Do you not wish your #LinkedIn data to be used for training their #GenAI models? Here is the link to opt out: https://lnkd.in/ehqW4eGn For more on this topic, see article by Cassie Kozyrkov.

19

2 Comments

Link to #Linux interview making the rounds on his thoughts around #AI. Even if we achieve #AGI (pick your own definition), the idea that #LLM AI's or LLM-based AI coding assistants or agents will replace human jobs in any short time frame is fundamentally predicated on 1 (human #behavior) training data includes an understanding of human roles and responsibilities in relation to each other how human "agents" interact to achieve complex volumes of work product. This information is somehow code in a structured way for it to be included. 2 (human #workflow) training processes can measure the quality of inputs as drafts, experts works, working copies, rough drafts, an experts' scratch notes, a verbose low level engineer whose uses variables names which are quite authoritative, the quick and dirty hack around branch, etc.. and so forth in order to objectively calibrate the models towards "better information" and "how information evolves". I'm not aware of any huge focus on org behavior on open source collaborations as contrasted to corporate projects, or joint ventures between institutions. Any have anything good send me. I don't see LLM's as currently able to generate new more optimal workflows (we have to train them to do that specifically, and they have to be decent at it), we're at the stage where current LLM's can [when wrapped up in automation logic] can recreate localized human workflows (solving test questions, code review/basic enhancement flows etc..). There's most certainly a few iterations on reasoning and just after it before systems exhibit truly divergent behavior. (Not saying there isn't a ton of short-term commercial potential in many, many areas). As someone who understands neural nets and low level machines intimately (I built my 1st kernel module as a tween, and built my own exotic neural net nets nearly 20 years ago. I look forward to participating in how the space unfolds. What I will say is that the hype is based on investor selection bias towards storytellers who want to play venture as it is currently played.. which may not be optimal. I have my terms and opinions. Here's a free forecast, I definitely see the current agent space as branching away from AI-wrapper / site builder towards advances in human computer interaction and human-human workflows brokered by computer... advances which are required for truly divergent systems to emerge. Definitely not ready to announce yet, but, the stuff I/we have been working on for a long long time will be looking for beta users later this year -- it is ready when it is ready -- DM me if you're into this stuff and don't mind making a radical dent in the space. #disent ✌ https://lnkd.in/eFtyU7a5

2