Harshal Dedhia

What does marketing in a post-cookies world look like? https://lnkd.in/gN7BPfjQ #datascientist #dataanalyst #cfo #cmo #productmanagers #founders #cofounders #cxo #ceo #cpo #bizops #revops #operations #b2bsaas #marketing #marketingmanagers

1

Stepping up to the Stage, Stepping into a New Age! Come join the conversation on my Substack!

10

Some random thoughts on the big smashing news the past two days: - Databricks buys Tabular - Snowflake open sources Polaris Catalog The two are very very related. #2 is highly likely a direct response to #1. 1. Ali is a force of nature, willing to make huge moves I don't think a lot of people saw the Tabular acquisition. I obviously wasn't in the rooms where this was decided, but I see this as a very very aggressive move by Databricks to win the fight for Tabular. Perhaps an advantage Databricks has over Snowflake by not being a publicly traded company? They can throw crazy numbers around? 2. Databricks has the upper hand but not for long When an Enterprise asks Snowflake what to use, Snowflake recommends Iceberg. Databricks, as of today, is neutral to Iceberg or Delta Lake. I think it's a temporary advantage, as Snowflake Polaris will have no choice but make Delta Lake a first class citizen. That said, Snowflake has no control over Delta Lake OSS project so they're on the backfoot. Delta Lake remains a more closed off open-core Databricks-only project, especially compared with Iceberg. 3. Both Snowflake and Databricks want better more open interop The fight is for the data lake service layer (that's essentially what a catalog is). However, the service layer needs to be fully interoperable with the ecosystem. I think we're going to see 3rd parties benefit from this competition, and, by proxy, customers using those 3rd parties.

211

30 Comments

💡 AI Adoption at data.world: Aligning with Industry Trends and Leading in Experimentation 💡 As generative AI gains momentum, enterprises are navigating its integration with a mix of excitement and caution. At data.world, we’re thrilled to see our approach aligning closely with key industry trends—and in some areas, leading the way. From robust training programs and active experimentation to meaningful adoption metrics, we’re fostering a culture that doesn’t just keep up with AI advancements but drives them. 📈 In a recent report by AI at Wharton and the GBK Collective ( https://t.co/LEPWQ69uoY), our own journey mirrors the broader trends they highlight, with data showing our teams are actively embracing AI tools like ChatGPT at levels well above industry averages. Curious about how we’re achieving this? Read on to see how data.world is setting the standard in responsible, innovative AI integration. #AI #GenerativeAI #Innovation #dataworld #AIAdoption #FutureOfWork https://lnkd.in/gWdVu_ZH

6

2 Comments

(I've spent more time than I'm willing to admit working on this post. Hope you get as much satisfaction from reading it as I get from publishing 😅) At Motif, the problem we've heard from almost every potential customer is that they want to understand what drives the successful and unsuccessful outcomes in their product. To reliably answer these causal discovery questions, we developed an entirely new approach that combined ideas from 1) causal inference and 2) sequence models currently powering the generative AI systems. The result is a methodology for automated causal inference that scales to thousands of cause-effect hypotheses, allowing users to efficiently find and prioritize the mechanisms for developing and prioritizing product improvements. In our early case studies, this approach is incredibly powerful and I am excited to see what we learn as we scale this out to our customers.

416

17 Comments

Unleashing Productivity with GenAI! GenAI helps enhance our product’s functionality and usability. Imagine richer databases and more meaningful insights at your fingertips—GenAI is making that possible! But that’s not all. We’ve built a cutting-edge RAG framework, enabling us to get answers faster and more efficiently than ever before. This is just the beginning of how GenAI is transforming the way we work. Stay tuned for more exciting developments! 🚀  #GenAI #Innovation #AIInAction #AI #Productivity #ProductDevelopment 

72

I have a great time spinning up our own delivery data in Omni, to the point that it deserved a write up on how easy is to use and the instant insights that we get from it! Check it out in our blog!

21

1 Comment

The data warehousing space continues to evolve with the GA of MotherDuck data warehouse. MotherDuck's POV seems to be that many companies don't really have "big data" and don't need the complexity of distributed system architectures (and associated costs) of a Snowflake or Databricks DW. Curious to hear from data warehouse practitioners whether their company's scale fits in DuckDB. https://lnkd.in/gbKFVT7r

6

🍜Anybody up for some ramen? I am excited to share that I am back in the garage, building again (it’s actually a house in the Berkeley Hills). We are taking the wraps off of Supercog AI, a new startup focused on solving Application Integration using LLM powered agents. My co-founder in this new venture is a great friend of mine (and a former co-founder partner), Alex Osborne. Alex and I have worked around the application integration space for a long time, and we are unrealistically excited about applying the GenAI stack to this problem. Not many people relish the task to get some bits to move from system A to system B. But it’s a critical job to be done, and that’s led to a lot of brittle scripts, a myriad of inflexible “no code” tools, and a lot of hand crafted SQL. But the power of the Large Language Model, trained on a huge corpus of information, offers a remarkable tool to solve this problem. The LLM you use today already knows the APIs of hundreds of popular systems. It knows the SQL dialects of every major database. It understands the semantics and data schemas of many popular SaaS systems. It understands much of the specific domain in which your business operates. I like to call the AI revolution “the last platform shift”. That may turn out to be hyperbole, but I definitely believe we are in the very early stages of seeing what this new stack can enable. I know we are hardly the first folks to claim that “amazing stuff is coming!”. But this is why Alex and I have decided to focus on a real and hard problem. If we can prove that LLMs can power a new way to solve this problem, it will be the first successful *new* approach in 15 years.. We will have much more to share about this new platform soon. In the meantime, if you’re interested in getting a peek - or even better if you have application or data integration tasks that you would like some help with - please reach out because we want to help. #genai #startups #backinthegarage

163

37 Comments

We just launched "partial ordering mode" for BigQuery DataFrames! Many thanks to my teammates, especially Trevor B., Qiuyan(Chelsea) Lin, Wu Jiaxun, and Sandeep Karmarkar for helping make this a reality. To use it, set the bigframes.pandas.options.bigquery.ordering_mode = "partial" option. This prevents BigQuery DataFrames from creating a deterministic ordering and sequential index. This can have a huge effect on performance. In my detailed explanation of the feature (https://lnkd.in/gxkJY-B6), I show an example of where this has a 4,000,000x speed up in terms of bytes scanned. Without this option set, BigQuery DataFrames creates a sequential index over all rows to mimic pandas behavior. On large tables this can get expensive, as it forces a full table scan, unless row and column filters are provided as arguments to read_gbq(). Partial ordering mode allows BigQuery DataFrames to push down many more row and column filters. On large clustered and partitioned tables, this can greatly reduce the number of bytes scanned and computation slots used. Partial ordering mode is currently in Preview. Is this feature helpful to you? Is there some functionality you need that is missing? The BigQuery DataFrames team would love to hear from you. Share your feedback at [email protected], bit.ly/bigframes-feedback, or by filing an issue at the open source BigQuery DataFrames repository (https://lnkd.in/gVFreXei). (Image shows a visualization of the PyPI public dataset, created using BigQuery DataFrames and the new partial ordering mode.)

33

2 Comments

Thanks Tom Goldberg for hosting me on your fantastic podcast, Vision to Venture! It was great to catch up on #AI , #datascience and #ml , the value of unconventional career paths, #agile and AI and so much more !! Love what y'all are doing!

12

1 Comment

Go from RAGs to AI-Driven Applications. I’ve already had multiple meetings this week (and it’s only Tuesday morning) helping clients transition from initial MVP AI RAG (Retrieval-Augmented Generation) concepts to robust, production-ready AI-Driven Applications. The full AI Application Lifecycle is coming into view for my team and for our clients. Our collaborative teams include Data Scientists, Software Developers, and Product Owners. 🔎 We also include AI Observability experts to ensure real-time governance of probabilistic GenAI outputs. 🔍 👩‍⚖️ 👩‍💻 👨‍🔧 Additionally, Product Users and subject matter experts play a crucial role, providing essential feedback to refine and optimize AI-Driven Applications for specific use cases. 👩‍⚕️👨‍💻👨‍🌾 Join me tomorrow at O'Reilly as we dive deep on Guardrails, Evaluation Metrics, and User Feedback,  key components of the mature AI-Driven Application development lifecycle. AI Observability: Frameworks for Evaluating Production LLMs Date: Wednesday, June 12 Time: 9 AM - 12 PM PT https://lnkd.in/dJSuUh2m #AI #MachineLearning #DataScience #AIDriven #Innovation #TechTalk #AIObservability #ProductDevelopment #UserFeedback #LLMOps #MLOps

38

3 Comments

Check out my PyData talk: "Crafting Impactful Dashboards for Your Clients"! https://lnkd.in/gi87hv2m #DataViz #analytics #datavisualization #plotly #python #datascience #diagram #figma

9

Want some cool general purpose AI tech that has nothing to do with LLMs? Check out The Forecasting Company They’ve built a “predict any time series” machine.  As an example - A big theme park chain can invest in detailed demand prediction ML, so they can predict how many visitors they will get and optimize staffing, maintenance, food production etc.  A small independent tourist attraction couldn’t, until now. To test the tech, I downloaded total hospital emergency admissions for England, excluded 2024 from the data I sent to the model and asked it to predict 2024 admissions. The result, not bad (see graph).  Significantly better than just using last year’s number as a proxy.  Could I have achieved the same or better with a few hours messing about with the data in excel, yes.  Could I achieve much better by modeling it all properly with a custom ML model and tons of clean custom features, definitely.  But it took 5 minutes, the model doesn’t require retraining (by me), and I got a decent response. My prediction, LTSMs (large time series models) are going to be a big deal, and this is just the first one I’ve seen. AI disclosure: No generative AI was used to create this post.

32

6 Comments

I'm diving headfirst into research around the impact of data quality on ML/AI with Christopher Bosley, and I'm curious to hear from YOU!  Do you ever struggle with inaccurate or missing data?  Struggling to understand the value of good data quality?  Do you want to know the impact of data quality on your ML algorithms? Share your biggest data quality pain points in the comments! If you know anyone passionate about the impact of data quality on ML/AI, particularly those interested in research, please tag them or send them my way!  #DataQuality #AIResearch

6

3 Comments

❄️ Snowflake Is in Talks to Buy Reka AI for $1 Billion Reka supports multimodal input (image, video w/ audio, text, and documents such as PDFs)

2

🚀 🚀 🚀 Roe AI helps manage your S3 unstructured data: you can now query S3 files directly with SQL. Today, AI teams have wasted too much time wrangling with raw S3 to make multimodal data and metadata work together. Figuring out which dataset to train which model is just a HOT mess. We simplify this process by providing a GUI that allows you to use SQL to organize the unstructured data as SQL tables. Combining with Gen AI agents, generate metadata for those raw data at scale Sign up here https://app.roe-ai.com/ to get your $50 free credits towards GPT-4o, Gemini pro 1.5 and more! Checkout my previous posts about using AI to cut soccer videos and analyze PDF documents) https://lnkd.in/dkgBf7Tu https://lnkd.in/dFDRRiMc

48

2 Comments

At my current company, Abound, we have a constant need to explore and develop insights about the user sentiment and decipher (at scale) whether a piece of text reflects positive, negative, or neutral emotions. Looking for advise on whether you ended up utilizing a rules based engine or did you utilize an AI based framework for classification of conversations and why. Here's how we landed on a rules based framework: https://lnkd.in/g-qGPW-s

1

I love the insights. We're starting to see a new landscape of enterprise search in data. And to your point, I think today's search is even more crazily limited in data / lakehouses (like, come on, regex matches are still the only supported SQL search query in year 2024)

13