Matt Fornito

Leaders have poor visibility into data, analytics & AI expenditure. That's a problem when they get asked how much more they need (Agentic AI & GraphRAG?), or a threat to teams and leadership when asked to cut spending in 2025 (LLM prototypes? BI reports?). ⚠️ Leaders blind to their spending allowed inefficiency and lack of strategic alignment to creep in. But how would they know what they don't know? 💡 I've learned from Gartner clients in objectively leading organizations, they regularly examine and BENCHMARK spending across multiple categories of #data, #analytics & #AI work and that this one activity facilitates smarter spending and better biz outcomes. Why? Without this it's hard to identify performance improvements, justify investments, optimize resources and shape strategy and operating model against your PEERS. BENCHMARKING empowers chief data and analytics officers, those at the head of data, analytics and AI teams to compare their spending, staffing and organizational design with real peers, of a similar size and industry, not just theoretical or hypothetical ratios and models. So... Jishnu Mohan, JC Martel, PhD, Alan D. Duncan and our team of quants have gone and built something for our clients to fix this. 🛠️ It's an interactive tool. It guides you to enter some key metrics then it automagically creates informative, comparative views of your org.'s D&A function against your peers. Out pops Gartner insights! This benchmark answers: 1️⃣ How does my D&A spend compare to my peers? - D&A total spend (CapEx & OpEx) - Spend on resources (personnel, software, PaaS & external services) - Spend on functional activities (data curation/enhancement, data governance, analytical data mgt. & platform dev, transactional data mgt.) - Spend on mgt. & emerging capabilities (D&A mgt., AI tech, spend supporting AI) - Strategic spend (run, grow, transform) 2️⃣ How does my funding model and org. structure compare to my peers? - Decision making for funding (model, ownership, budget) - D&A org. structure (hybrid, centralized, federated) - Job title of the head of D&A and primary responsibilities 3️⃣ How does my staffing plan compare to my peers? - Total D&A employees (FTEs and contractors) - Staffing by functional activities and emerging capabilities - Staffing by roles (current vs. next 12 months, by FTEs & contractors)  4️⃣ How do my peers measure value from D&A use cases? - Metrics for measuring value from D&A use cases We've made this exclusively for clients who are CDAOs (see link in comments). If that's not you 😢 don't fret, we're publishing a summary of findings this week 🙂 for other clients, and I'll summarize highlights in another post - for everyone. Thank you to my colleagues who contributed: Mark Beyer, Francisco Javier De los Santos Ramos, Jorgen Heizenberg, Soren Krohn, Timothy Kropp, Brian Foster, Rita Sallam & Eric Stegman. 👍 Like if you think this sort of tool is useful. For CDAO clients ➡️ link in comments.

59

4 Comments

Are you interested in... - Data viz tepid takes? - Slightly humorous example graphs? - Pie chart hate? If you answered "yes" (or even "maybe") to any of the above, then you should check out my latest GovLoop article on data viz!

6

A/B Testing myth: “Longer tests = more sample size = more power” The shared post gives a great formal intuition why “longer tests = more sample size = more power” sometimes is NOT true. Must read for every experimenter! For more information on this topic also check out section 3.4 in this paper (https://lnkd.in/evBypYtu) by Ron Kohavi & collegues or this post (https://lnkd.in/euE33Th7)

101

26 Comments

What can you achieve in a day? These workshops address the 2 largest gaps that I see - the "soft" or invisible skills we assume analysts have (such as effective requirements gathering) and the advanced visualization skills that go beyond do-and-do not for data visualization. Systems thinking helps zoom out the camera lens to draw in the larger picture. So many dashboards assume they are the lone answer to a given problem. The reality is that most people are touching a variety of disparate systems to do their jobs. This course addresses that gap and others. After years of presenting on how to Put Functional Aesthetics to work, I have a workshop for you that addresses the gamut of how we visualize - from products to dashboards to infographics for the public, this course helps people enrich their practice with cutting edge information from both research and practice. DM me for more information and please feel free to share widely. #Workshops #DataViz #Training #Analyst #AnalystSkills #DataFam

26

A company recently came to us with a huge challenge: over 200,000 open-ended survey responses collected over 5 years, all needing to be analyzed and synthesized into insights within weeks. 😅 This problem isn’t unique to big co. Government agencies and orgs conducting large scale medical or consumer research face similar overwhelming data challenges. The manual methods we use for small studies won't work. But AI alone isn’t enough. While AI can process data quickly, it often misses the nuance and depth needed for meaningful insights. Some specific limitations with an AI-only approach: ✨ AI focuses on high-frequency terms, often missing emotional undertones, contradictions, or deeper themes. ✨ It struggles with tone, sarcasm, and complex emotional feedback. ✨ AI might group irrelevant data together because of its reliance on statistical frequency. ✨ It can reflect the biases of the datasets it’s trained on, often missing underrepresented voices. ✨ And the key limitation: handling extremely large datasets. Most consumer platforms require data to be processed in smaller batches due to memory and processing limits, leading to fragmented, disjointed insights. That’s why we launched a new service. 💁‍♀️ It's a perfect blend of our backgrounds in consulting, research expertise, and the technology we’ve built with Notably. I truly think our product is the best qualitative analysis tool on the market and you won't find anyone more passionate about synthesis and big data challenges than our team. 🥰 I put together this blog post to elaborate on the challenges and break down how we approach this challenge with our blend of tech & services. If you have a mountain of data you need help analyzing quickly, our team is here to help! https://lnkd.in/dQ3-z6D8 #uxresearch #qualitativedata #consultingservices #AI #datanalysis #AIandTech #HumaninTheLoop

16

1 Comment

#eli5 #contextualretrieval AI systems like chatbots need to retrieve relevant info to give the best answers. Normally, developers use a method called Retrieval-Augmented Generation (RAG) to fetch info from large databases. The problem? RAG sometimes misses important context, leading to wrong answers. A approach called Contextual Retrieval fixes this issue by adding extra context to each chunk of info in the database. This improves accuracy and makes the AI smarter, retrieving better info and improving responses by up to 67%. How does it work? 1. Contextual Embeddings: It adds extra, specific context to each chunk of text so the AI knows exactly what it’s about, reducing confusion. 2. Contextual BM25: An old-school technique that finds exact keyword matches, great for things like specific error codes or terms. For developers, this means less failed searches, faster results, and better customer support, legal advice, or any task requiring complex knowledge retrieval.

9

2 Comments

How startups in Data/AI/ML build their GTM teams? Insights from 500,000+ employees reveal a consistent pattern. We've analyzed data from over 500,000+ current and former employees across more than 1,000 AI/ML companies. Here are the key findings: 1) Initial focus 60% of the first GTM hires were in sales and support, followed by 37.5% in marketing, and just 2.5% in engineering. (*Note: engineering in this case would refer to solutions/sales engineers) This makes sense. Sales is the obvious first stop when one thinks about selling a product. BUT, an interesting trend we've seen in recent years is a commitment to standing up the marketing flywheel EARLY. More and more companies are pledging resources to brand, content, and product marketing very early on (hence the 37.5% slice here). 2) Expanding the team: Subsequent hires increasingly favor sales and support, with 80% for the second hire and 90% for the third, emphasizing the critical role of building strong sales teams. ***Here's where I think this data can be slightly deceiving: We capture data across companies that range from seed stage all the way to industry Goliaths like Snowflake. Yes, all these companies were startups at one point. BUT, building a GTM function in 2024 is not like it was in 2010. The motion back then, an thus the personnel needed, are different. We are forgetting about the very first GTM team member: The Founder! In the early days, the founder is the seller. Some other things to consider - 🤌Marketing typically takes a longer time to marinate. Unless you are crushing a paid media lead capture approach (which is less likely to be effective in today's market), founders should expect marketing to be a longer term investment. ☎️Sales might lead you to quicker results, but headcount costs can rack up quickly. 🕵️The very first GTM hires (whether sales or marketing) will need to to focus on more than just immediate revenue. They will: a) help you refine the product, b) help you define your ICP, c) help you design a GTM motion that resonates with your prospects. Thus, these are special people. They are special hires. Check out these two posts I made that breakdown the activities of early GTM hires at Snowflake and Hubspot: Hubspot: https://lnkd.in/gTCzxfZr Snowflake: https://lnkd.in/gyMzVQez Thanks to Jason Saltzman from Live Data Technologies for the crucial employment data (visit livedatatechnologies.com for more). Interested in a deeper dive? DM 'DATABASE' for a customizable data export. With this database's support, we've assisted numerous successful early-stage startups—backed by giants like Insight Partners, Sequoia Capital, Andreessen Horowitz, and more—in securing top-tier talent.

49

10 Comments

SAS Visual Analytics Report Builders, how do you go about choosing the best chart? When choosing the chart to best display your data and advance your data story, you need to think about the audience and the type of data you want to display. Have a look at Nicole Ball's article for some guidance and tricks. #SASusers #SAS #SASviya #dataviz #SASva #data #datastory https://lnkd.in/eqFBS3hP

20

2 Comments

"We need them to generate new business, lead teams, manage partnerships, AND be hands-on." Sound familiar? It’s because these are the MVPs of Data Consultancy—Principal Consultants and Team Leads. I’ve hired many over the years, and one thing is clear: these professionals are essential for driving growth, maximizing client value, and building resilience for the future. Looking back at 2024, many businesses felt the pinch from over-hiring and focusing solely on delivery in 2023. It made sense at the time—so much work, so much revenue on the table. But now, as we head into Q4 2024, the focus has shifted. It’s a race to secure top-tier Principal Consultants and Team Leads—the people who will be pivotal to thriving in 2025. Here’s why: 📖 Experience: They bring deep expertise and thought leadership. 🔎 Business Development: Masters at uncovering opportunities and closing deals. 🤝 Client Relationships: They don’t just manage accounts; they elevate them. 👊 Leadership: Mentors who inspire teams to perform their best. 💡 Innovation: Problem-solvers who push boundaries and drive evolution. But here’s the challenge: attracting these individuals takes a headhunt approach. They’re busy, deeply embedded in their roles, and often not actively exploring the job market. If your business isn’t actively building strategies to attract them via internal TA or externally (ON Data Staffing), you may already be falling behind your competition. #DataHeadhunter #DataRecruitment #DataConsultancy #Databricks #Snowflake #DataEngineering #DataAnalytics

8

Serious question about synthetic participants. If we assume that synthetic participants can fill out a survey good enough to stand in for real participants, why do we need the survey at all? If we have a question, and we're saying that we believe the AI + data (synthetic participants) knows the answer, then why have the AI pretend to be 500 people, take a survey, analyze the survey, and get at the answer through a very labor intensive market research pipeline? Why not just ask the AI what the answer is?

6

15 Comments

Curious about integrating AI into your research workflow? If you're relying on basic summarization tools or just using ChatGPT, you're missing out on AI tools designed specifically for research. With Notably, you can collaborate with AI in 4 super specific and targeted stages of your research process: 1. Data Cleanup & Summaries: Clean up transcripts, scrub PII, and summarize calls using tailored AI prompts for usability sessions, stakeholder interviews, and more. 2. Analysis: Instruct AI how to highlight and tag raw data so you can uncover patterns and emerging insights, faster. 3. Synthesis: Experiment with clustering data with AI using various frameworks, tags, or by participants for rapid visualization. 4. Insight Generation: Generate insights using AI-driven prompts crafted to promote divergent, generative thinking. Join us this Weds at 12pm EST to break down each of these 4 steps in Notably. Plus, we’ll be launching a new feature during our session (what could go wrong?!) where you can choose from pre-made prompt instructions to highlight & tag and cluster data. RSVP to secure your spot! PS: If you can't make it live, still RSVP and we'll share the recording https://lu.ma/597pwyj3 #AIResearch #UXResearch #QualitativeResearch #AI #AIEvent #UXR #AITools

6

1 Comment

The physical copy of my book is still pending. But right now my e-book is ready for pre-order! https://lnkd.in/g_ik8Vgs I'm extremely excited for this release. A year and a half in the works of thinking about it. Editing on it. Having other people read it. As mentioned previously this is the first book of many addressing different aspects of workplace and home cultures that are driving true effectiveness and innovation. I ask you to share this! Give feedback, be unfiltered. This represents a 60% solution. As a community let's build this concept to at least 90%. I'm a fan of physical books so I'm more excited about the release of that one. But it's still pending review. Summary from Amazon "Default Laziness: Organizational Ideals of Perceived Effectiveness explores the intricate dynamics within organizational systems that inadvertently promote inefficiency and stifle innovation. Authored by a seasoned veteran of the Air Force with extensive experience in innovation and systems management, this book delves into the concept of "default laziness" — a systemic inertia fostered by over-simplified processes, excessive outsourcing, and a deep-seated aversion to risk. Through a series of insightful articles, the author examines how entrenched practices and the consolidation of control into the hands of a few external 'prime' entities can lead to organizational stagnation and dependency. The book critiques the prevailing organizational philosophy that equates streamlined operations with effectiveness, challenging readers to reconsider how modern organizations manage and mismanage their resources." Daniel Ramsden Jared Pino John Teichert, Brigadier General (ret)Chad Hardesty John Hawley Colin Dziadaszek Raymond Chalifoux, MSOL, PMP, LSS Nathaniel Pahlow Ian Eishen #innovation #book #defaultlaziness

20

13 Comments

I think there are overlooked implications of auto model selection in ChatGPT, which seems to be rolling out slowly. First, this can massively impact OpenAI’s profitability if it can direct more queries to smaller models without adversely affecting quality. Second, this can prevent people leaving ChatGPT for other apps—like those running local models—if it’s just easier to stay in one app and get a faster model or smarter model automatically, rather than use small local models for some things and switch to ChatGPT for others. It’s even possible that ChatGPT could in future run local models on properly equipped devices for the easiest queries, which would completely avoid hitting OpenAI’s servers for those queries. So, it’s a small feature, but I think one with big implications.

15

7 Comments

🎉 And the results are in... 🎉The best type of Analytics Engineering job opportunity is 🥁 Last week, I posted a poll to uncover the most attractive type of Analytics Engineering role. Here's what you all had to say: 1️⃣ The first data hire — capturing the logic around the business and setting up the data foundations and stack: 37% 2️⃣ Joining an established data team with a mature stack — typically growth or replacement hires: 34% 3️⃣ Joining a data team that's new to the modern data stack or Analytics Engineering: 28% Are you surprised by these results? 🤔 I'd love to hear your thoughts in the comments! 💬 #DataAnalytics #AnalyticsEngineering #DataCareers #PollResults #DataTeam #ModernDataStack

6

Adrienne Murphy discusses an outstanding example of data leadership by the Massachusetts Department of Early Education and Care. Transcript: Adrienne Murphy (Massachusetts Department of Early Education and Care): First, in terms of understanding the pre-K landscape, the CUSP network has really helped us understand, in this three and four-year-old space, who's being served across the mixed delivery system, which has been especially important when our governor, the Healey Administration, announced a new Gateway to Pre-K Initiative, which included a proposal to deliver universal, high-quality preschool access for four-year-olds in all of our Gateway cities, which are our sort of mid-sized urban centers that are sort of the regional economic anchors across the state. And the cost model has been able to help us to benchmark sort of where are we now, who's being served across our licensed and funded early education care providers, across our public K-12 system, across our private school system, to be able to understand sort of the full landscape of who we're currently serving. And, by virtue of who we're currently serving, also, who are we not serving? Sort of what's the delta between where we are now in pre-K access for four-year-olds across the state, but really particularly in these gateway cities? And then sort of, what's the delta between where we are now and the universal access that is proposed in the gateway to Pre-K initiative? So, this is, you know, really one instance in which being able to know who are we currently serving, who are we not serving, and where are we not serving children has become really key for us and for Massachusetts as sort of the policy design process. Click here to learn more: https://team3si.com/ #communityofpractice #dataforgood #earlychildhooddata #acf #ecids

1

Here's a tip for how data people can build strategic relationships with execs quickly that I like to call: "The Book Play 📚" Ask a business person you work with for the book that has most influenced the way they think about their subject area. Then read the book! This is transformational in two ways. 1. You're able to do a much better job because you have shared context. If a CMO thinks about marketing a certain way... By understanding how they think about marketing, you're going to do a much better job supporting their goals. Not to mention you'll probably learn more about marketing in general which is also important if you do data work for marketing! 2. You're proving you care about what they think and are willing to go beyond what most people do to help them. Most people don't do stuff like this. By executing the book play you set yourself apart, and show that you're willing to invest in them. When you ask - DON'T ask for their favorite business book, because you'll get generic recommendations like "How to win friends and influence people." You'll lose the benefit of understanding how they think about their specific subject area. Here's a talk track you can use: "What book has most influenced the way you think about marketing? I want to read it so I can do a better job supporting the team." Bonus points: Get a few recommendations of people to follow on social media who have smart things to say as well. Gives you a constant stream of topics you can bring up in 1:1's to ask "I saw this post, seems relevant to us, what do you think?"

23

3 Comments

SEAL is a new model benchmarking service from Scale AI that fills a big gap: an additional objective, third-party, trustworthy way to compare models. People distrust many benchmarks because they worry that models are optimized to perform well on them—including by training on the underlying data (like cheating on a test by memorizing the questions and answers in advance). The LMSys Chatbot Arena (https://chat.lmsys.org) addresses this by having humans evaluate model output in blinded head-to-head tests. The new SEAL leaderboard (https://lnkd.in/gdCZzRnz) does something similar, but uses "thoroughly vetted experts using domain specific methodologies" to do the head-to-head evaluation. Sounds much more expensive, but they just raised $1 billion, so I think they can afford it.

1

Forrester's End-User Experience Management Wave is LIVE! Our most in-depth analysis of this growing market. Check out the blog post for some high-level analysis. #Forrester clients can access the full report. Interested in learning more? Join my webinar on August 20th. As always, a massive thanks to the vendors, customer references, and Forrester ecosystem for all the hard work put into this report. https://lnkd.in/gzx6GN3K #Digitalemployeeexperience #DEX

112

2 Comments