Casey Karst

✨ If you have Databricks, do you need Snorkel AI? If you plan to customize #LLMs with your own enterprise data the answer is an easy yes. You need high quality data to build production-quality AI, but most unstructured enterprise data is inaccurate, incomplete, and inconsistent. Often you have a lot of data for common scenarios but very little for high-value edge cases (obviously!). Good news: Snorkel Flow integrates seamlessly with Databricks at every stage of the AI development workflow, from accessing data & models to deploying production-quality applications via Unity Catalog. For a guided walk-through check out this 5min demo and learn more at https://lnkd.in/gTiQ4wD8 #AIInnovation #ArtificialIntelligence #dataintelligence

31

7 Comments

Matillion x Cortex components now GA 👀 "We are excited to announce that Matillion now supports all of the major Snowflake AI capabilities, in a no-code, easy-to-use platform providing a complete and powerful AI solution that integrates with existing data workflows." Get all the details here (or swing by booths 1517 and 2502 at #DataCloudSummit) https://lnkd.in/eQ3dHm7i #AI #DataEngineering #DataEngineer

72

2 Comments

Databricks disclosed that they are expecting to cross $2.4B in revenue run-rate growing ~65% Y/Y by H1’25 (Jan FYE). This was alongside top-tier metrics, including 80%+ GMs, 140%+ NDR, and a multi-product story where ~50% of customers use 6+ products (incl. their Data Warehouse product which generated $400M in revenue up 4.0x Y/Y). Given their accelerated growth, Databricks is among the few scaled software companies, alongside hyperscalers, capitalizing on the AI boom. Equally noteworthy is their market share gain against competitors. Considering their growth trajectory, we explored various scenarios to determine when Databricks will surpass Snowflake in scale. The question is not if, but when, and we believe it is sooner than most anticipate.

1,010

26 Comments

Apache Iceberg, an open source table format for large-scale analytics, improves upon the limitations of traditional storage solutions. It allows for fine-grained control over data, enabling features like schema evolution, time travel, and transactional support—all crucial for modern data architectures. https://bit.ly/3JZJ6hp #InfluxDB #OpenSource

16

Lately, we have observed a transition from Cloud Databases(Snowflake etc) to Iceberg OpenLakehouse format. Ambaricloud team is working on Snowflake to Iceberg Automation and end-user training. Why: Low-cost computing. SF is still a favored engine for Gold layer data. Less stickiness. More engine options to run on Iceberg tables. Cloud storage Parquet files Iceberg table format Catalog Status: Polaris -- Started Walking. Hopefully, it will Jog or Run soon! UnitCatalog -- Infant Glue -- Safe bet for now. Comments? Snowflake Cloudera Dremio Upsolver Databricks

8

1 Comment

Hey everyone, def check this out. If you have a startup putting data together with AI, it is 🍌 🍌 🍌 🍌 to not build it on a system that unifies the two. You need a framework like this. And get a prize!

76

2 Comments

Unlock the Power of Data Lake Table Formats with Alex Merced, Developer Advocate at Dremio! 📊 Join Alex Merced as he delves into "The Who, What, and Why of Data Lake Table Formats." 📅 Save the Date: https://lnkd.in/eFZ6m5HV 🔍 Key Learnings: - Explore how data lake table formats handle large datasets. - Learn how these formats optimize query performance on extensive data. - Understand the importance of data durability and recoverability. - Discover how these formats ensure data is auditable and compliant. Don't miss out on this opportunity to level up your data analytics game! 📈 #DataAnalytics #DataLake #Dremio #TechTalk

4

If you are ingesting data from Kafka/Kinesis into Snowflake you should watch this short video by Charles Tan and Shawn Gordon on how you can use DeltaStream to shift left your workloads to reduce cost and latency! #Flink #Kafka #Snowflake #ShiftLeft https://lnkd.in/gnZ45MTa

13

Meet Llama 3.2 on Databricks: faster and now multi-modal. The Llama 3.2 release pushes the frontier of enterprise #GenAI with new smaller models for latency + cost-sensitive use cases and larger multimodal models for visual understanding. Customers can easily get started tuning the Llama 3.2 models securely and efficiently on their data. Available today in Databricks Mosaic AI. 

9

In our 2024 State of Data Security Report, only 50% of organizations said that their data security strategy is keeping up with AI's rate of evolution. To learn how Databricks Unity Catalog and the Immuta Data Security Platform enable security and governance for cross-platform #AI models, meet our team at Databricks Data & AI Summit, and check out this blog: https://ow.ly/wQXz50RvVtj Schedule your onsite meeting today: https://ow.ly/7LZA50RvVti #DataAISummit #DataSecurity

17

As part of Databricks' LakeFlow Council, I've had the opportunity to see the work that is happening being the scenes on LakeFlow Connect, Pipelines, and Jobs. From what I see, "Pipelines" is what DLTs should have been, and MUCH more. About 80% of what I'd like to see appears to be getting implemented. Future appears bright for this one thus far! PS: Shared this with Bilal Aslam's permission.

90

10 Comments

Unlock the power of Amazon product review analysis with Apify and Snowflake Cortex! This guide walks you through scraping reviews, loading data into Snowflake, and summarizing insights. #DataIntegration #Snowflake

10

We have out-of-the-box support in LlamaIndex for Snowflake SOTA arctic-embed models (which outperform other leading embedding models in benchmarks) - it's a one-liner through our Hugging Face integration! And in ~4 more lines of code, you can build a RAG indexing pipeline over your data with these embedding models. Check out the screenshot below for how to get up and running with this - we might have more detailed benchmarks soon, stay tuned 💡

213

3 Comments

In this clip, Josh Miramant explains how Databricks Serverless is transforming data ops, making life easier for analysts and engineers alike. 🌐 Now available on Azure, AWS, and soon GCP, this multi-cloud solution offers flexibility and speed like never before. To watch the full interview, see: https://lnkd.in/e9i833_J Topics include: 1. Differences between Databricks Serverless and traditional clusters 2. How serverless enhances cost efficiency and resource management 3. The impact on non-technical users and the ease of use for analysts 4. Potential limitations of serverless for specific workloads 5. The future direction of Databricks Serverless and its multi-cloud support #databricks #serverless #cloudcomputing #dataengineering #analytics

12

With its announced partnership with Anthropic’s Claude AI models, Snowflake is creating a lot of buzz in a technology landscape that is defined by the break-neck speed of acquisition, innovation and evolution. But is the November 20 announcement just another spin around the AI hype cycle, or something to get excited about? We’ve put together some thoughts on three immediate impacts organizations and developers can expect for Enterprise AI applications in the Snowflake Data Cloud. #snowflake #datatoaction #claude #aireadiness #aistrategy https://lnkd.in/g8M_eKCm 

9

1 Comment

Will you be at the Databricks Data+AI Summit? Don’t miss the SqlDBM team this week. Visit booth 20 to learn about our all-in-one data modeling platform! 🚀 #Databricks #DataModeling #SqlDBM

24

2 Comments

CData is back in SF, this time for Databricks #DataAISummit 2024! ✈️ Visit Booth 96 to connect with the CData team and explore how you can access data from across hundreds of applications and systems in Databricks. Come say hi! 👋

41