Dustin Vannoy

A common misconception with AI and machine learning: “all of the resources, infrastructure and technology" it takes to get started. With Databricks ML Flow, that’s simply not the case. In the video ⬇️ Kenny Shaevel breaks down how Databricks makes it easier to manage and track your models while ensuring full transparency, traceability, and governance—without the headache. Kenny covers everything from: - Simplifying Machine Learning Implementation - Utilizing AutoML for Quick Data Evaluation - The Importance of Feature Engineering - Leveraging Feature Store for Reusability - Governance and Traceability with the Model Registry - Seamless Integration and Application of Models Excited for more? Kenny will be diving even deeper into Databricks in our upcoming live stream series. Keep an eye out for details coming soon! 👀 #machinelearning #databricks #dataanalytics #MLFlow #AutoML #featurestore #dataengineering #techinnovation

5

Navigating the Databricks landscape can be challenging, even for seasoned data engineers. Our resident expert Robert Leach wrote about the common mistakes people make, and offers practical advice on how to: 👉 Tailor your clusters to drive for performance and cost efficiency 👉 Ensure consistency, compliance, and reliability with proper data governance policies 👉 Boost your productivity by mastering the Databricks Command Line Interface 👉 Implement robust access controls, encryption, and regular security reviews 👉 Make the most of the platform’s capabilities You’ll want to bookmark this post! 🔖 https://bit.ly/45DZqhY

15

New Blog: Using Apache Hudi & Iceberg tables in Databricks with Apache XTable (Incubating) 🎉 In this hands-on blog, I show how we can use XTable to interoperate among the 3 open table formats within Databricks environment. I explore a use case where a team of data scientists run ML experiments using MLflow on top of Hudi/Iceberg tables by consuming these tables as Delta Lake. Some key points to consider & how XTable helps: ✅ While predominantly focused on Delta tables, this team also would like to access Apache Hudi/Iceberg datasets from other teams to enhance their ML models. ✅ In the past, the data platform team has either duplicated these datasets to rewrite them as Delta Lake or conducted one-off migrations (costly). ✅ Data Platform team's goal: the ability to read table formats other than Delta, should be seamless & dependency-free, without the need for additional configurations. ✅ Irrespective of whether the source table is Hudi, Delta Lake or #ApacheIceberg, users should just be able to read the dataset as if it were a Delta Lake table using spark.read.format(“delta”).load(“table_name”) Link in comments. #dataengineering #softwareengineering

123

6 Comments

Llama3 on Databricks. Databricks is changing the game with making open models easily usable.

1

Streaming Tables and Materialized Views now supports Row filters and Column Masks. If you want to get started quickly with RW and CM : https://lnkd.in/eaNNJpci #Databricks

60

3 Comments

🚀 Now supporting Databricks compute! 🚀 You can now use your Databricks clusters for generator training and synthetic data generation, alongside Kubernetes. This update unlocks new possibilities, providing more flexibility for your workflows. #syntheticdata #databricks #ai #generativeai #mostlyai #datascience

10

1 Comment

⚖️ Tech Comparison: Redshift vs Druid https://lnkd.in/e3ZCD3xb Like {hashtag|\#|Snowflake}, {hashtag|\#|Redshift} was built for traditional BI and infrequent reports. Not having multi-level indexing and time-based partitioning means inefficient compute with more data processed than necessary. When the analytics use case requires high performance on large data sets or concurrency requirements and streaming data, then Apache Druid is ideal.

10

Did you miss this awesome demo Hugo Lu did with Youssef Mrini over at Databricks 🏘 In Orchestra you can trigger and monitor Databricks Workbooks, Workflows, data quality tests, and lots more. Our users also leverage Orchestra's overarching monitoring and alerting capabilities to link upstream jobs like data ingestion (e.g. ADF) through Databricks into downstream jobs like Reverse ELT with Hightouch or Power BI dataset refreshes. Check it out here 🔗 https://lnkd.in/dU2grpum #dataengineering #databricks

7

1 Comment

Predictive AI workflow with Kumo + Databricks Kumo operates directly on raw Databricks tables, utilizing Unity Catalog for data governance and security, generates predictions at scale, and writes them back to your #Databricks warehouse. The data warehouse native deployment for Databricks (in private preview) ensures that no data is stored on disk in a Kumo owned environment and any data leaving Databricks is transformed, encoded, and deleted after use. Reliably and securely create predictive AI to make predictions without data ever being stored outside your Databricks warehouse. Learn more and request a demo here: 🔗 https://lnkd.in/dYREGfvE. #PredictiveAI #MachineLearning #DataScience #GNN #LLM #Databricks #AIInnovation #BusinessIntelligence

13

ChaosSearch + Databricks 📈 Deliver on the best of Databricks (open Spark-based data lakehouse) and ELK (efficient search, flexible live ingestion, API/UI) via ChaosSearch on Databricks. ChaosSearch natively supports Delta Lake and Spark on Databricks, creating a seamless experience in the Databricks platform. ChaosSearch, as a Databricks Technology Partner, brings ELK use cases to Databricks ecosystem enabling centralized log and event analytics for observability and security. Learn more: https://lnkd.in/dSAXRgEF

3

Check out this awesome demo Hugo Lu did with Youssef Mrini over at Databricks 🌊 🏘 In Orchestra you can trigger and monitor Databricks Workbooks, Workflows, data quality tests, and lots more. Our users also leverage Orchestra's overarching monitoring and alerting capabilities to link upstream jobs like data ingestion (e.g. ADF) through Databricks into downstream jobs like Reverse ELT with Hightouch or Power BI dataset refreshes. Check it out here 🔗 https://lnkd.in/dU2grpum #dataengineering #unifieddatacontrolplane

11

2 Comments

Snowflake releases arctic-embed models, a family of 5 text embedding models that range from 23 to 334 million parameters. The models are the result of search knowledge, and research and development that Snowflake acquired last May via Neeva. 💵 The models outperforms retrieval performance, having one-quarter of the estimated parameters and one-third of the dimensions compared to Open AI. 📊 https://lnkd.in/gjJy9Yte Some more insights on how the technical achievements are detailed in this blog 🦾https://lnkd.in/g_aYgwrH

2

1 Comment

I presented about Apache XTable's architecture last week at LinkedIn's Big Data Meetup. For context, XTable is an omni-directional translation layer on top of open table formats such as Apache Hudi, #ApacheIceberg & Delta Lake. It is NOT ❌ a new table format! Essentially what we are doing is this: SOURCE ---> (read metadata) ---> XTable's Model ---> write into TARGET We read the metadata from the SOURCE table format, put it as a unified representation & write the metadata in the TARGET format. * Note that we are only touching metadata, not the actual data files (such as #Parquet) with XTable. Let's breakdown its architecture. XTable’s architecture consists of three key components: 1. Conversion Source: ✅ These are table format specific modules responsible for reading metadata from the source ✅ They extract information like schema, transactions, partitions & translate it into XTable’s unified internal representation 2. Conversion Logic: ✅ This is the central processing unit of XTable ✅ It orchestrates the entire translation process, including initializing of all components, managing sources and targets, among other critical things 3. Conversion Target: ✅ These mirror the source readers ✅ They take the internal representation of the metadata & maps it to the target format’s metadata structure Read the paper (link in comments) for details. #dataengineering #softwareengineering

199

5 Comments

Blogged: Interoperability between lakehouse table formats with Apache XTable (Incubating). Scenario: Dremio -> Apache Hudi ❌ Dremio -> Apache Iceberg ✅ There are 2 analytics teams in an organization. Team A uses Hudi to manage some of their most critical, low-latency data pipelines. Team B focuses on ad hoc analysis, BI, and reporting, using Dremio's compute engine with Iceberg. ❌ Challenge: Team B wants to do analytics using data from the 2 different datasets within the organization. To do so, Team B has to use the dataset generated by Team A (stored as a Hudi table) and combine it with their dataset (the Iceberg table) ✅ Solution: Interoperability between the formats with Apache XTable. Detailed reading link in comments. #dataengineering #softwareengineering

105

4 Comments

The latest news posted by Jason Ganz is about dbt Labs's integration with Snowflake as a Native App. This is a significant development for data enthusiasts and professionals alike, combining the strengths of dbt Cloud’s discovery and semantic capabilities with Snowflake's robust architecture. Here's why this is a game-changer: Seamless Integration: The dbt Snowflake Native App allows users to access powerful dbt features directly within the Snowflake UI. This means one login to unlock a wealth of insights from your dbt projects. Enhanced Data Discovery: With dbt Explorer, business analysts can now navigate data products and deliver reliable insights effortlessly. Column-level lineage, cross-project lineage, and rich metadata provide a comprehensive view of your data. Natural Language Querying: The new Ask dbt feature, currently in beta, lets users ask questions in plain English and get SQL-generated answers. This is powered by Snowflake Cortex AI, making data interaction more intuitive and accessible. Improved Data Trust: Orchestration Observability ensures data freshness and job success visibility, helping maintain trust in your data and streamline issue resolution. Easy Access and Procurement: The app is readily available on the Snowflake Marketplace, and you can use your Snowflake committed spend for procurement, simplifying the purchase process. This integration represents a growing trend towards making sophisticated data tools more accessible and user-friendly, reducing the barriers to gaining actionable insights. The collaboration between dbt and Snowflake showcases how leveraging combined strengths can lead to more efficient and effective data-driven decision-making. #DataAnalytics #Innovation #dbtCloud #Snowflake #BigData #AI #DataDriven #TechTrends

2

We recently started a new LinkedIn Live series for everything Apache Hudi and Open Lakehouses 🎉 In the very 1st episode, Dipankar went over the fundamentals of Apache Hudi and showed hands-on examples of running transactions with compute engines like Apache Spark. If you missed the live, here's the recording: https://lnkd.in/dm-24JzC Check it out and share with the Hudi community! #dataengineering #softwareengineering #lakehouse

15