Matteo Pelati

OpenPipe Introduces a New Family of ‘Mixture of Agents’ MoA Models Optimized for Generating Synthetic Training Data: Outperform GPT-4 at 1/25th the Cost Quick read: https://lnkd.in/gfxdiDT5 OpenPipe’s MoA models have excelled in rigorous benchmarking tests, achieving notable scores on LMSYS’s Arena Hard Auto and AlpacaEval 2.0. The MoA model scored 84.8 on Arena Hard Auto and 68.4 on AlpacaEval 2.0, indicating its superior performance in generating high-quality synthetic data. These benchmarks are critical as they represent challenging user queries that test the robustness and adaptability of AI models. The MoA model has been benchmarked against various GPT-4 variants in real-world scenarios. Results showed that OpenPipe’s MoA model was preferred over GPT-4 in 59.5% of the tasks evaluated by Claude 3 Opus. This is a significant achievement, highlighting the model’s effectiveness and practical applicability in diverse tasks encountered by OpenPipe’s customers...... OpenPipe

42

2 Comments

#ObsessivelyEnthusiasticInvestor #UnreasonablyEnthusiasticInvestor #IrrationallyEnthusiasticInvestor at REFASHIOND Ventures: The Industrial Transformation Fund. We look for the people we believe are doing the best work in Industrial Transformation & Supply Chain Technology; Attempting the impossible. Then we go all-in on them. With obsessive and unreasonable enthusiasm. Join us. #IndustrialTransformation #SupplyChain #Innovation #Technology #Startups #VentureCapital #DemandTheImpossible #UnreasonableEnthusiasm #ObsessiveEnthusiasm #IrrationalEnthusiasm 

6

Large language models are wonderful at ingesting large amounts of content & summarizing. Uploading an academic paper and I can pester it with an infinite list of questions & it will respond with equally infinite patience. In comparing the two most recent Microsoft earnings calls, Claude highlighted: - faster than expected Azure growth (29% vs 27%) - AI contributing 8% of Azure revenue up from 7% - higher CapEx spending & greater capacity constraints for data centers - much better commercial bookings growth : 17% vs 13% Excellent analysis. Benn Stancil described LLMs as great averagers of information. But what if I don’t want the average? If I seek the data point two standard deviations out? If I’m channeling my inner Anthony Bourdain & I’m seeking the fermented shark or cobra heart within my query? At Google, we ranked web pages & ads with many signals. An engineer taught me that of them found rarity of a word across a set of documents. For example, if there’s a document containing 10 instances of the word ghotiacross a collection of documents, where the average document has 0 ghotis, that document is likely the best to answer a search about fish. I haven’t found a way to goad an LLM to produce the rare result. Why did one user decide to use a particular piece of software when 10 others did not? Maybe I haven’t learned how to prompt an LLM well. Ideogram launched a feature called Magic Prompt that expands a basic prompt into richer instructions that elicit better results, marrying the language of a user & a computer. A user might write : A boy with a dog in a park. Magic Prompt replies : A heartwarming scene of a young boy playing with his happy, wagging dog in a lush, green park. The boy has a playful smile on his face while he tosses a bright yellow ball for his furry friend to fetch. The park is filled with beautiful flowers and trees, creating a serene atmosphere for this bonding moment between boy and dog. Magic Prompt surprised me in three ways. First, unlike Google search where terse queries tend to work very well, LLMs relish verbosity. Less Hemingway & more Faulkner. Second, maybe there’s a prompt for rarity. If you know what it is, please share with the rarity shibboleth. Third, I wonder what other prompt techniques are out there that have yet to be discovered to wrest the best from an LLM. This technique is called TF-IDF. It means Term Frequency Inverse Document Frequency.

21

6 Comments

Quick tip - DDL for vector indexes https://lnkd.in/gwtNtE_r

38

LLMs and Optimization: An Unbeatable Combo!

19

Catching up on some of the great presentations from #PEPR24 presentations, and came across Damien Desfontaines’ talk on synthetic data. I obviously disagree with lots of it, but I think Damien generally makes his points in good faith. So I’m going to (try to) do the same in response. The part of his talk that stood out to me as most obviously incorrect and uncharitable was the discussion of the incentives for improved privacy measurements in synthetic data. I’ve included his graphic below as a summary of his argument that there’s no commercial incentive to improve empirical privacy evaluations of synthetic data for context. Sharing three thoughts: 1) If there’s no commercial incentive for better privacy evaluations with synthetic data, differential privacy companies shouldn’t exist. The complexity and form factor of differential privacy imposes enormous incremental friction to configure and use relative to synthetic data. If there’s no commercial case for better synthetic data privacy metrics where none of that friction exists, then how could there possibly be commercial demand for more friction and better privacy metrics — the asserted value of differential privacy? As far as I know, Tumult Labs isn’t a charity, so someone there must believe there’s a market for stronger privacy guarantees — we agree with that, even if Damien doesn’t 😂 2) That’s the theoretical argument, but I’m a synthetic data guy, so I’ll make an empirical argument, too: prospects and customers care deeply about the rigor of privacy tests for synthetic data and it is a buying criteria in *every* deal in our funnel. For *privacy* use cases (more on this in the next paragraph), buyers are not just asking for better metrics; they’re demanding it. My data on this is MUCH better than Damien’s — since he doesn’t actually sell synthetic data to customers — and he’s completely wrong. 3) Despite their website copy, most synthetic data companies are delivering to non-privacy use cases *at least as often* as they’re delivering to privacy-enhancing analysis use cases. Synthetic data is a really useful technology for data quality work, test data management, and anonymization. And in practice, most synthetic data platforms are most heavily used for data quality work (imputing NULLs, class rebalancing, dataset augmentation, etc.). Privacy is — at best — a secondary criteria for these use cases, so the basic privacy metrics are sufficient. At Subsalt, we work exclusively on anonymization use cases and we win deals on the basis of our privacy work. We’re living proof that there’s a business case for more rigorous privacy work in synthetic data. I don’t expect this to solve the eternal (and cursed) debate about empirical vs. theoretical privacy metrics, but I do think it’s worth clarifying that if you want to know about the structure of a market — in this case the market for synthetic data — you should listen to someone that knows at least something about that market.

13

5 Comments

To start, I have to concede that the quote from our website (yes, Subsalt is the anonymous company in Damien Desfontaines’ post; I don’t think he even tried that hard to make it anonymous 😂) overstates the real risks of re-identification in synthetic data. The copy will be changed, because it actually is a good example of sloppy privacy messaging. Mea culpa. But despite being a pretty good dunk, it’s not outcome determinative on the issue we’re actually discussing. The relevant question is: is there market demand for more rigorous privacy metrics in the synthetic data space. The answer is unambiguously “yes.” Customers are not satisfied and they justifiably don’t believe the most commonly used metrics add up to “anonymization.” So why haven’t most synthetic data vendors adjusted their “privacy scores” to accommodate this dissatisfaction? The most commonly referenced companies in synthetic data go to market with freemium versions of their product that can be adopted and used by end-users without a multi-stakeholder purchasing process. The users that actually adopt these tools tend to use them disproportionately for *non-privacy* use cases. And more stringent privacy tests are not neutral to the UX for these users: they’re a source of friction and complexity. So the versions of these synthetic data products you can try off-the-shelf stick to basic privacy metrics that don’t introduce friction. The dirty little secret in synthetic data is that VERY few synthetic data workloads in production are premised on sufficient data anonymization through synthesis. That’s because synthetic data deals where anonymization is the primary use case have traditionally died in the risk, privacy, legal, and compliance departments *because those stakeholders want more rigorous privacy guarantees* that conflict with the needs of the non-privacy users of synthetic data platforms. If every large organization on earth had deployed a synthetic data platform for privacy use cases in spite of this weakness, you’d be able to conclude that there’s a lack of demand for more rigorous privacy work in the space. But all of the synthetic data vendors (combined) have a relatively tiny penetration into the market. The biggest source of today’s “non-consumption” (e.g. the fully uncaptured portion of the addressable market) is the misalignment between the standard synthetic data form-factor and privacy measurements and buyer demand for anonymization. Despite the sloppy copy on our website, that is the latent demand we hear from prospects and customers every day and we take the problem very seriously. I’d be happy to share more about our privacy metrics and approach, because I share the deep concerns and commitment to providing rigorous privacy preserving tools so organizations can do more with sensitive and regulated data — even if I don’t share the skepticism of all empirical privacy metrics.

9

2 Comments

🤖 MetaGPT Redefines Multi-Agent Collaboration 🤖 📄 arXiv 𝗣𝗮𝗽𝗲𝗿: "MetaGPT: Meta Programming for a Multi-Agent Collaborative Framework" by Sirui Hong et al. During my talks, people usually ask me where I expect to see a boom in the incoming months. If Sam doesn't find AGI in the short term 😂, my answer is clear: Agents, Agents everywhere! So let's take a look to this paper about Agent Collaboration. 📝 𝗔𝗯𝘀𝘁𝗿𝗮𝗰𝘁 𝗦𝘂𝗺𝗺𝗮𝗿𝘆 Take a look to MetaGPT, a game-changer in multi-agent collaboration. By leveraging Meta Programming techniques, MetaGPT optimizes collaboration efficiency through Agent Roles, Workflows, Structured Messages, and Efficient Communication Protocols. Its outstanding performance on benchmarks like HumanEval and MBPP sets a new standard, particularly excelling in software development tasks. 𝗧𝗟;𝗗𝗥 🧠 MetaGPT introduces cutting-edge Meta Programming techniques for enhanced multi-agent collaboration. 📊 Achieves unparalleled performance on industry benchmarks like HumanEval and MBPP. 👨‍💻 Excels in software development tasks with high task completion rates. 🤝 Incorporates structured messages and efficient communication protocols for streamlined collaboration. 👾 Ablation study underscores the significance of diverse agent roles in boosting performance. 𝗙𝗼𝗿 𝗺𝗼𝗿𝗲 𝗶𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻 👇 📜 ArXiv Paper: https://lnkd.in/dt7XuM3Z 📪 Andrew Ng post on Multi-Agent Framework: https://lnkd.in/d5pmYcpB 👩‍💻 MetaGPT Github repository: https://lnkd.in/dQa7T4wM 👏 Kudos to Val Andrei Fajardo for the amazing visualization of the pager, it's pure magic for the eyes!

53

4 Comments

Architectural decisions for CoveredCalls.AI back/forward-testing: (1) DuckDB, combining Polygon S3 aggregates into single parquet file for analytics. https://lnkd.in/ga7AFTYY (2) Time-series in Supabase with Timescale-extension, ingesting into tables and using indicators. #Finance #AI

4

Semantic chunking vs. fixed-size chunking: Which delivers better results in Retrieval-Augmented Generation (#RAG) systems? 🤔 In our latest blog, Renyi Qu and Forrest Sheng Bao, engineers at Vectara, challenge the hype around semantic chunking. Through systematic evaluation across RAG tasks, they uncover surprising insights into performance, computational costs, and real-world practicality. 🔍 Key Takeaways: ✅ Fixed-size chunking often outperforms semantic chunking in real-world scenarios. ✅ Computational costs of semantic chunking don’t always justify the benefits. ✅ Focus on embedding quality for bigger gains in RAG performance. 👉Check out the full blog here: https://lnkd.in/gZ9M7eYD

15

OpenAI's o1 (newest model) is already available on Wordware (YC S24) Try it vs Anthropic's Claude-3.5 Sonnet (used to be best at reasoning) Click the link below to ask a difficult question and see how they do! Try this for example: If you stack 5 identical cubes in a tower, and each cube is 3 inches tall, how tall is the tower if you remove the middle cube?

65

7 Comments

Sonnet 3.5 has been makingn waves. Personally I have found that perhaps it lacks breadth of knowledge ( not able to comprehensively answer many questions as well as openai) but its definitely faster , mostly gives better code and provides projects and artifacts features which are reason enough for switching to it for techies doing using it for programming tasks.

1

This is a win for #sustainability. In effect, ML applications may better leverage GPU capabilities, saving more time and energy 🚀

1

🚀 Unveiling the Ultimate Kafka + RisingWave Cheat Sheet! 📊✨ Ready to power up your real-time streaming workflows? 🤔 Dive into our comprehensive Kafka + RisingWave cheat sheet! 📃 🔑 Key Highlights: ▶️Set up Kafka + RisingWave in minutes ⏱️ ▶️Real-time data ingestion and transformation 💻 🔰Join our amazing community 👉 go.risingwave.com/slack #Kafka #RisingWave #StreamProcessing #DataStreaming #RealTimeData #CheatSheet #DataEngineering

12

𝐀𝐠𝐞𝐧𝐭𝐢𝐜 𝐖𝐨𝐫𝐤𝐟𝐥𝐨𝐰: 𝐏𝐢𝐨𝐧𝐞𝐞𝐫𝐢𝐧𝐠 𝐭𝐡𝐞 𝐅𝐮𝐭𝐮𝐫𝐞 Recently I happened to see this nice video ( 𝘩𝘵𝘵𝘱𝘴://𝘸𝘸𝘸.𝘺𝘰𝘶𝘵𝘶𝘣𝘦.𝘤𝘰𝘮/𝘸𝘢𝘵𝘤𝘩?𝘷=𝘴𝘢𝘭78𝘈𝘊𝘵𝘎𝘛𝘤 ) by Andrew Ng in which he talks about Agentic workflows. In Language Model (LLM) development, we provide a zero-shot prompt and receive an answer. We may review it and add more prompts to get it to a desired shape. This is exactly what the agentic workflow does. Basically, 𝘢 𝒃𝒖𝒏𝒄𝒉 𝒐𝒇 𝑨𝑰 𝒂𝒈𝒆𝒏𝒕𝒔 𝒘𝒐𝒓𝒌 𝒕𝒐𝒈𝒆𝒕𝒉𝒆𝒓 𝒊𝒏 𝒂𝒏 𝒊𝒕𝒆𝒓𝒂𝒕𝒊𝒗𝒆 𝒘𝒂𝒚 𝒕𝒐 𝒑𝒓𝒐𝒅𝒖𝒄𝒆 𝒂 𝒅𝒆𝒔𝒊𝒓𝒆𝒅 𝒐𝒖𝒕𝒑𝒖𝒕. Moreover, he even claims the output produced by agentic workflow is much better than the output produced single LLM which might sound obvious but what surprised me was, as per the “𝑯𝒖𝒎𝒂𝒏𝑬𝒗𝒂𝒍” coding benchmark, agents-based approach beating plain LLM by quite a margin (see the picture below). The plain 𝐆𝐏𝐓 3.5 was at 48% but 𝑳𝑫𝑩+𝑹𝒆𝒇𝒍𝒆𝒄𝒕𝒊𝒐𝒏 (𝘈 𝘓𝘢𝘳𝘨𝘦 𝘓𝘢𝘯𝘨𝘶𝘢𝘨𝘦 𝘔𝘰𝘥𝘦𝘭 𝘋𝘦𝘣𝘶𝘨𝘨𝘦𝘳 𝘷𝘪𝘢 𝘝𝘦𝘳𝘪𝘧𝘺𝘪𝘯𝘨 𝘙𝘶𝘯𝘵𝘪𝘮𝘦 𝘌𝘹𝘦𝘤𝘶𝘵𝘪𝘰𝘯 𝘚𝘵𝘦𝘱-𝘣𝘺-𝘴𝘵𝘦𝘱) fared at impressive 95.1% with the same GPT 3.5. The plain 𝐆𝐏𝐓 4 was at 67% but 𝑨𝒈𝒆𝒏𝒕𝑪𝒐𝒅𝒆𝒓 (𝘔𝘶𝘭𝘵𝘪-𝘈𝘨𝘦𝘯𝘵-𝘣𝘢𝘴𝘦𝘥 𝘊𝘰𝘥𝘦 𝘎𝘦𝘯𝘦𝘳𝘢𝘵𝘪𝘰𝘯 𝘸𝘪𝘵𝘩 𝘐𝘵𝘦𝘳𝘢𝘵𝘪𝘷𝘦 𝘛𝘦𝘴𝘵𝘪𝘯𝘨 𝘢𝘯𝘥 𝘖𝘱𝘵𝘪𝘮𝘪𝘴𝘢𝘵𝘪𝘰𝘯) fared at whooping 96.3% with the same GPT 4.  You can get more information about the evaluation and the models at 𝘩𝘵𝘵𝘱𝘴://𝘱𝘢𝘱𝘦𝘳𝘴𝘸𝘪𝘵𝘩𝘤𝘰𝘥𝘦.𝘤𝘰𝘮/𝘴𝘰𝘵𝘢/𝘤𝘰𝘥𝘦-𝘨𝘦𝘯𝘦𝘳𝘢𝘵𝘪𝘰𝘯-𝘰𝘯-𝘩𝘶𝘮𝘢𝘯𝘦𝘷𝘢𝘭. He also mentioned the following are the 4 design patterns that are emerging in the agentic workflow. 1.    Reflection 2.    Tool use 3.    Planning 4.    Multi-agent collaboration If you are interested in learning more, instead of me explaining each of these patterns, I would encourage watching the video :) #ai #agent #agenticworkflow #gpt

23

We benchmarked Apollo vs WunderGraph vs Mesh vs Grafbase GraphQL Federation gateways. Here are the results (and my take on the benchmarking): CONTEXT: Last year, we launched Grafbase GraphQL Federation (built in Rust) from scratch. To see how it performs compared to others, we’ve started benchmarking it against other providers. Here’s the full benchmarking setup: - All gateways had the same CPU and RAM resources, limited by Docker. - We repeatedly executed a complex query using the load-testing tool K6, with a constant and ramping VU scenario. - The final response content is 88KiB big, merged from several subgraph responses. Now to the interesting part… RESULTS: Throughput: - Grafbase Gateway: 1502 req/sec - Cosmo Router: 867 req/sec - Apollo Router: 434 req/sec - Mesh: 300 req/sec Latency: - Grafbase Gateway: 57.9 ms - Apollo Router: 92.4 ms - Mesh: 127.2 ms - Cosmo Router: 308.7 ms (We’ve benchmarked more than just throughput & latency, the full blog post is linked in the comments below) MY TAKE: First, I’m happy to see that all the hard work we put into the Grafbase Gateway paid off. I believe it’s mainly due to two things: 1/ We betted on Rust to build the Grafbase Gateway right from the start – and that’s still paying dividends today. Some other core design choices also contributed to this result. 2/ As a company, we’re striving for a culture of performance. This goes beyond our engineering team, but definitely reflects in product performance. But what does this result mean from a business perspective? 1/ First, great performance helps you reduce your infrastructure footprint. For enterprise companies, we’re talking about billions of requests per minute (yes, minute). These seemingly small differences quickly turn into hundreds of thousands of dollars in infrastructure costs. 2/ It’s a better foundation for scaling. This is especially relevant for large enterprises. If you expect heavy load on your systems (because you want to scale), your foundation needs to be rocksolid. At this level, there’s no room for small performance issues. 3/ Better user experience Above all, better performance means a better experience for your users. I think I don’t need to elaborate on this. The effects of great (or not so great) user experience can be felt in *every* aspect of your company. – To be honest, I was also impressed by some of the results. The sustained latency vs. throughput metric showed that you can keep throwing requests at the Grafbase Gateway and get guaranteed great latency. If you’re interested in the full benchmark, check out the link in the comments. All that’s left to say: Kudos to our amazing engineers at Grafbase. – PS - We tried to be objective. If you feel like we should benchmark other metrics or see ways to improve - comment below, file a complaint or open up a PR. PPS - If you encounter a case where Grafbase doesn’t provide this level of performance, we’d be happy to look into it for you.

40

8 Comments

You do realize that 2GB of ram means 2B bytes? For 3B params? And you need multiple bytes for each token? You can quantize all you want - LLMs will never be reliable in limited memory save for narrow purpose-build cases, that’s just physics. (Some would say they will never be reliable, period. )

3

3 Comments

LLMs are not vast, infallible databases. As models grow larger so does their tendency to confidently state falsehoods (hallucinations). Matei Zaharia, co-founder & CTO of Databricks, warns against treating LLMs as internet-scale encyclopedias and proposes we focus on what they excel at:  → Reasoning → Interpreting language → Understanding context For founders building AI applications, treat LLMs as powerful reasoning engines and not fact repositories. Build your knowledge base separately, feeding the LLM with trusted up-to-date information through retrieval mechanisms and tool-calling. This approach lets you harness the LLM's strengths in language analysis and generation while sidestepping the risks of its unreliable memory. Building with AI? Share your approach to balancing LLM reasoning power with factual accuracy in the comments. Let's hear it!

137

22 Comments

🥁 Llama3 is out 🥁 8B and 70B models available today. 8k context length. Trained with 15 trillion tokens on a custom-built 24k GPU cluster. Great performance on various benchmarks, with Llam3-8B doing better than Llama2-70B in some cases. More versions are coming over the next few months. https://lnkd.in/dNZx72FJ

19,194

293 Comments