Gergely Dányi

𝑳𝑳𝑴-𝒂𝒔-𝑱𝒖𝒅𝒈𝒆 🧑‍⚖️ 𝘓𝘓𝘔 𝘌𝘷𝘢𝘭𝘴 & 𝘵𝘩𝘦 𝘪𝘯𝘴𝘪𝘥𝘦 𝘴𝘵𝘰𝘳𝘺 𝘰𝘧 𝘩𝘰𝘸 Twilio Segment 𝘣𝘶𝘪𝘭𝘵 𝘊𝘋𝘗 𝘊𝘰𝘱𝘪𝘭𝘰𝘵 𝘵𝘰 𝘢𝘤𝘤𝘦𝘭𝘦𝘳𝘢𝘵𝘦 𝘮𝘢𝘳𝘬𝘦𝘵er 𝘸𝘰𝘳𝘬𝘧𝘭𝘰𝘸𝘴 3𝘹 These days I'm spending an irrational amount of time thinking about evals. It started with a conversation months ago with Jaimal Soni, when he was showing me an absolutely 🔥🔥 demo of what he's been building in stealth. I couldn't believe what I'd just seen. "What's one of the hardest problem you had to solve?" I asked. "Evals." Evals are the way to assess (ahem evaluate) LLMs and systems built with LLMs. In software engineering, you'll hear a lot about test coverage and unit tests — little tests to verify that your code is doing what you expect. But generative AI & LLMs are a little different: unlike your deterministic code, models are stochastic, meaning you never really know what the output will be until it's generated!Ask chatGPT the *same* question and you will get *different* answers. This is where LLMs get their magic, but also creates problems when you're asking an LLM to solve discrete tasks. As we've been building out CustomerAI & Generative Audiences, we needed to think about this problem from the start. Generative Audiences allows marketers to write natural language to generate new audiences, without having to navigate a sophisticated audience-building UI. We’ve been blown away by the results of – customers are experiencing a 3x improvement in median time-to-audience creation, and world-class feature retention rate IF the audience generation works on the first attempt. So evals are 🔑 Ankit Awasthi, Salman Ahmed, Alfredo Láinez Rodrigo, & James Zhu came up with a really clever solution to ensure that CDP Copilot is generating audiences that are accurate & represent the intent of the prompt. But it wasn't easy...and today we're sharing how we used synthetic data & LLM-as-Judge architecture to accelerate marketer workflows 3x.

90

8 Comments

A Data Geek’s Take on GenAI: Building a data product people love is not an easy task. Curious how LLMs and RAG are changing the game? (2 minutes read) https://lnkd.in/d8Jiap96 #DataExperience #GenAI #PersonalizedData #DataStrategy #RAG

8

“I have to say that you always have to keep learning. Unfortunately, I know it's comfortable to be able to do the job that you're doing all the time, but technology is different, it changes and it costs a lot of money and if you're in that domain where you find yourself and your skill set getting a bit older you can get pretty lost in your career pretty quickly so it's important to keep up.” Wise words from Hubert Dulay during our latest conversation from the Unapologetically Technical podcast. Indeed, constant learning is important no matter how comfortable we think we are in a certain job, and it’s through learning that we get to keep up with technology’s changes. Watch the full episode here to learn more: https://lnkd.in/dFE2rU4z

2

I often talk to enterprise marketers about AI testing -- i.e., using reinforcement learning to personalize all aspects of marketing communication to each customer -- and how it's much more efficient and effective than A/B testing. A frequent reply is "we don't do A/B testing; we do multivariate testing, which is much better." Then, I have to gently disagree with the speaker, and explain how it's not much better -- in fact, it has many of the same drawbacks as A/B testing. Essentially, it's parallelized A/B testing, where the tradeoff is increased testing speed of multiple variants, due to the parallelization, but reduced power (how much of an effect you're able to detect) due to the splitting of the audience into many small groups. The best way I found to explain this is to describe the full range of experimentation approaches: A/B testing, multivariate testing, multi-armed bandits, and contextual bandits. For the full story, check out my blog post: https://lnkd.in/eYDwry-4

60

1 Comment

I’m super excited to speak at the Turing Fest on the 9th of July. My talk will start with a simple question: How can you measure the value of a dashboard or a report? Answer: The value of a dashboard or a report is measured by the useful actions it prompts. But sadly, the reality is that after investing hundreds of thousands of dollars in your data stack, data is not as useful as it should be, and people don’t act on it. My presentation will focus on helping you fix that! By the way, the title I’m using is from my favorite tweet from Gwenyth: https://lnkd.in/dQkxCkG5 Join my talk, or DM me if you haven’t got your ticket yet. I’ve got a discount code for you ;)

34

3 Comments

🔥 2-Min Survey: Help Fix the Insights Pipeline, Boost Your Next Review At Ara Platforms we're *obsessed* with the data -> insight -> action pipeline and removing roadblocks. On the data (insight producer) side, we want to make it easier to generate and publish useful insights. On the stakeholder (insight consumer) side, we want to make it easier to ask questions and get useful answers quickly. The process feels like a broken market—data people produce insights that don't get used while stakeholders don't can't get insights to important questions in a timely manner. If you're on either side of this equation I'd love to chat—please reach out via DM! If you can't it would mean a ton if you could fill out one of these quick Google forms. The end goal is to improve everyone's performance reviews Insight producers (data scientist, analysts, researchers): https://lnkd.in/emtMV6um Insight consumers (product managers, leadership, anyone who uses data to make decisions): https://lnkd.in/eEJ3Wse7

4

1 Comment

🚀 We've open-sourced TimesFM – a decoder-only foundation model for time-series forecasting 📈 Key Highlights: ✅ Large-Scale Training Data: Trained on a corpus of over 100 billion real-world time points, including Google Trends and Wikipedia pageviews. ✅ Zero-Shot Forecasting: Offers high-quality forecasts even on unseen datasets, without requiring fine-tuning or additional training. ✅ Outstanding Benchmark Performance: Achieved competitive results across a comprehensive suite of benchmarks. 🤗 Explore the model on Hugging Face and GitHub. I'm encouraged by the early feedback and look forward to seeing how #TimesFM will be used in real-world applications for forecasting and anomaly detection. 🙏 A special thanks to the talented research team behind this achievement: Abhimanyu Das Rajat Sen Yichen Zhou Weihao Kong

44

1 Comment

New from lytix ai! Agentic evaluations for your agentic workflows. Starting with a new, agent-driven approach to evaluating your RAG pipelines and a new evaluation metric - "Potential sources missed". If you're ready to try it out yourself, check out our docs (📑 https://lnkd.in/gDumEXxY) on how to get started. And here's some context on why/how we built it (📝 https://lnkd.in/gbkTCWYT) We leverage findings from "Agent-as-aJudge: Evaluate Agents with Agents" (Zhang et. al - link in comments), to solve 3 major problems we see teams struggling with in production. 1. In the real world, LLM-driven workflows are typically multi-step processes (vs. a single model receiving the users input, doing all the work in a single inference call, and producing an output). But while most evaluators let you evaluate the final output, they ignore the step by step process it took to get there. That's critical evaluation data you're missing, to understand WHY your outputs went 'wrong', and what specific steps in the process you need to improve. 2. How do you evaluate RAG pipelines without a 'source of truth'? So many RAG evaluators require comparing an output to some fixed 'golden dataset'. But in reality, maintaining that source of truth is half the battle! And what about if each of users has a separate source of truth that they're querying against? For example, consider NotebookLM from Google - everyone is querying their own documents they uploaded! If you're building a product with a similar dynamic, are you supposed to maintain a golden dataset for each of your users? All just for evaluations? 3. Many RAG evaluators can tell you the extent to which your model has used the chunks that it's pulled from the source of truth, in it's final answer. But what about testing if the agent grabbed the right chunks in the first place? What if there is important context in a chunk that the agent missed? Today, developers have 0 visibility into that part of the funnel. We introduce a new metric called "potential sources missed", to shine some light here.

23

4 Comments

At the Databricks Data+AI Summit 2024, I was captivated by the dialogue surrounding AI. What struck me most was the clear divide between Generative AI's perceived potential and the practical challenges of its implementation. Despite its immense promise, Generative AI still has a long way to go. 𝗞𝗲𝘆 𝗣𝗼𝗶𝗻𝘁𝘀: Generative AI models often don't meet the precise needs of industries like finance and healthcare. Approx. 85-90% of AI projects fail to reach production due to misalignment and poor infrastructure. Regulatory compliance is critical, especially in highly regulated sectors. Slick demos can be misleading; practical implementation needs a deep understanding of business needs. Incremental progress and continuous data quality evaluation are essential.

22

Unleashing the Power of Geospatial Analytics with Snowflake! 🗺️ A must-see demo from Volodymyr Bilonenko 🇺🇦🇪🇺 on how Snowflake's geospatial features are transforming how Vay visualizes and analyzes geographic data. Whether you're working in GIS, logistics, or retail, Snowflake's scalable, cloud-native platform is a game-changer for processing complex geospatial data at lightning speed. From mapping to real-time data visualization, Snowflake is helping businesses make smarter, location-driven decisions. 💡

53

2 Comments

We built a fully-automated #ConversationalAI Agent deflecting 10,000+ monthly customer support tickets – in 10 steps at Voiceflow. Here’s how we did it: - Step 0: Prep Help Center (Knowledge Base) - Step 1: Core Workflows (ie. Submit a Ticket) - Step 2: Full Prototyped Wireframes - Step 3: Transition Legacy Support (ie. Intercom bubble) - Step 4: API Calls & Data Sources (ie. Zendesk/Segment) - Step 5: Establishing Modalities & Channels - Step 6: Segment Knowledge Base(s) & Chunks - Step 7: Optimize Models & Prompts (L1-L4) - Step 8: Metrics, Analytics & Testing Suite - Step 9: Polish interaction & Conversation Design - Step 10: Final testing, prep for prod No jira. No agile. No sprints. No standups. We just focused on the user, putting the customer outcome first. ✅ 93% support ticket resolution rate ✅ First reply time: 308min → 1min ✅ CSAT increased to 93.8% ✅ Slashed cost/ticket from 31 USD to 16 USD, to net savings. And…it was filled with learnings while working in the cutting edge of LLMs focused on deriving value from #GenerativeAI – and you can do it too! A comprehensive end-to-end platform like Voiceflow, takes you from idea & design to Production in 10 days. One-tool, end-to-end – all with one PM & one Dev. How it works: 1. Obsess over your intended customer experience (both granularly and broadly) for all 10 phases. It’s the only thing you should do. 2. Define scope and user flow first. Then design it, prototype it, and push to beta-prod immediately. 3. Fix any hallucinations, chunks and KB approaches that arise for initial release. It's scary fast, and straightforward, focus all-in the tooling - the rest falls into place. Comment if you’re interested, and I'll send the overview deck to help you build your first automated AI Agent, or improve your existing AI #chatbot.

71

21 Comments

I post often about Stardog Voicebox, the only AI Data Assistant powered by a native enterprise knowledge graph platform. Today I want to tell you about performance increases in that platform. We care a lot, because our customers do, about aggregate knowledge graph queries, and the best benchmark for those is LSQB. Over the past few releases, culiminating for now in the forthcoming Stardog 11, our performance on this benchmark has increased 109%. That means more insights faster for our users and more wall clock time to add more value (about which more later). I’ll take that win-win any day.

40

Maxa | One Year Of Growth After Winning The 2023 Snowflake Startup Challenge Co-founders and Co-CEOs Raphael and Alexis C. Steinman of Maxa.ai sit down with "Data Cloud Now" anchor Ryan C. Green to provide an update on the progress their company has enjoyed since winning the 2023 Snowflake Startup Challenge and the work they are doing to automate financial and operational (ERP) insights for financial and business teams. #SnowflakeSummit #DataCloudNow

50

Build, test, deploy, activate/interact and manage with your business AI Agents based on OpenAI, AWS Bedrock, Google Vertex AI, Anthropic, Mistral, or custom LLMs using the Quickpath Low Code/No Code platform. #AIOrchestration #QuickpathPlatform #AIInnovation #langchain #llm #AIAgents #Openai #Chatgpt #Anthropic #Bedrock #VertexAI

11

This week we’re launching 10BY10 - a ten week series hosted by Avii, Agentiiv’s digital person. Every week Avii will be summarizing a complex report or whitepaper on AI and The Future of Work, and delivering a concise 2 minute video summary of the main takeaways and a more detailed written review with a link to the original report. The full workflow, including the summarization, video script writing and of course Avii herself is AI augmented. The objective is to share meaningful value through distilled insights that you can digest in under 10 minutes, and give you the breadcrumbs to learn more if you’d like to access the full report. This week we’ll start off with a comprehensive look at how marketers are adapting with the "2024 State of Marketing AI Report from Marketing AI Institute”. Check out the detailed written summary in Avii’s blog here:  https://lnkd.in/ghS2jA2v

80

26 Comments

#Churn prediction is one of the common tasks of "internal" data-science teams. Did you know that, on average, it takes 8 months to build and deploy a churn prediction model? In our latest blog post Itai, who's working with Forwrd.ai's customers on many of these models, has crafted an "how-to" guide for planning, building and activating a churn prediction model. 🔗 Link to the post is in the first comment. #DataScience #AI #Guide

27

1 Comment

At HData, we value our customers and their feedback. The number one question we kept hearing is, "How can we verify that the answers your Regulatory AI solution provides are accurate?". Realizing that asking people to read hundreds or thousands of regulatory pages to verify the accuracy of answers is not feasible, we set out to address this challenge head-on by building the world's first Regulatory AI solution that is not a "magical black box of answers"; instead, Citations empower users to see not only where the answers were sourced from but also why. Here's to AI improving quality of life 😎 #regulatoryai #regtech

14

🌟 "Ask Simply, Grow Quickly" isn't just a tagline—it's a data revolution in the making! 🌟 As Co-Founder of DeZign Bytes, I’m thrilled to share what truely drives us! For so many small to medium businesses, data insights often feel just out of reach, buried under complex code and queries. We wanted to change that. *Insert* the creation of ChattterBox, our AI-powered, zero-code tool!! With ChattterBox all it takes is a simple question to unlock your data’s story. 📊 Whether you’re a data pro or just getting started, you can now chat with your data, gaining insights without a technical manual in sight. Why “Ask Simply, Grow Quickly” matters and isn't just a cheeky tagline.... Ask Simply: Skip the jargon and dive right into your data! With ChattterBox, just type a question, and watch the insights roll in. Need help formulating your queries? Our AI prompts have you covered. Grow Quickly: When answers are just a question away, decision-making speeds up, and for small businesses, that can mean quicker pivots and growth opportunities without the steep cost of complex tools. It’s incredible to see the empowerment and confidence that accessible data insights bring. I’m excited to invite you to be part of this journey. 🌐 Curious? Join the “Ask Simply, Grow Quickly” movement with us. Sign up for ChattterBox, and let’s make data your favourite team member! 🥳 https://lnkd.in/gxCseVs9 #ChattterBox #DataMadeSimple #AskSimplyGrowQuickly #DataInsights #SaaS #NoCode #AI

1