Dustin Dorsey

👀 This is gold and super top of mind for so many enterprises. They want to modernize their Databricks + Snowflake investment so that they aren’t needlessly moving/duplicating data, aren’t double-dipping compute, and are moving towards the trend of open standards and table-formats. 💪🔥

59

2 Comments

⛑️☁️ Preview on how Epic Cogito Cloud on Microsoft Fabric will be able to work with Snowflake without copying data: "This will enable our joint customers to experience bidirectional data access between Snowflake and Microsoft Fabric, with a single copy of data with OneLake in Fabric. Organizations using both platforms will be able to do so more cost-effectively, rather than building pipelines or maintaining copies of data in each platform." - Christian Kleinerman ❄️ #interoperability #datacloud #apacheiceberg #apachextable #fabric #msfabric #epic #epiccogito #cogitocloud #clarity #caboodle #nebula #epicnebula #snowflake #onelake #aidatacloud #aiml #healthcare #healthIT #hcldr #hitsm #healthtech

21

4 Comments

The Nashville Data Engineering Group is hosting a special meetup with Fivetran and dbt Labs next week on Thursday, June 27th! This event is totally free and will be in-person at the Vaco offices in Brentwood from 5:30 to 7:30pm. This is an exciting opportunity to meet other local data professionals and learn more about a couple of the most popular tools in data engineering right now and the problems they are solving. For more info and to register please check out the comments! #NashDataEng #DataEngineering #NashvilleTech

42

3 Comments

I think todays announcement of Datavolo acquisition is a drop the mic moment for Snowflake in terms of building #data #pipelines. Quick research shows #Datavolo is an #AI driven Low/No code ETL tool based on #OSS Apache NiFi which can build both structured and unstructured data pipelines within minutes by simply users describing the pipeline they need in plain English. Imagine you asking "Build a pipeline that ingest JSON files from S3 bucket called 'XYZ' into a table called FOO in the TESTDB Database where the files may be compressed. Modify the date format in ColX from MM/YY to YYYY-MM" and this thing builds an entire #NIFI data pipeline workflow within seconds where you can make modifications later on if necessary. I have no idea how Snowflake plans to implement it but if this is close to how it may work in the product, it will be a total game changer for #dataengineering for #datawarehousing & #GenAI use cases. It will literally accelerate onboarding new data sources from weeks to minutes while allowing our customers to not just ingest + tranform/process structured & semi-structured data but also unstructured data as well for new GenAI use cases from hundreds of sources with built-in connectors. Needless to say, I am super excited & can't wait to get my hands on it once they incorporate it in to the product. God Speed!!! Check out these quick demos: Ingesting JSON to Snowflake https://lnkd.in/gJM7xXb2 Prepping Docs in Google Drive for a RAG application: https://lnkd.in/gQi_K4n9

638

24 Comments

With #IcebergSummit happening tomorrow online, I took a trip down memory lane by listening to the #DataFemme episode featuring Alex Merced of Dremio. This episode is packed full with information, not just about #Apache Iceberg, but about the best ways to get started as a contributor to #opensource projects. Alex is an expert content creator with extensive knowledge about Apache products and their role in technology. This is a must listen for everyone heading towards Iceberg Summit tomorrow! https://lnkd.in/guqYdzG2 Tabular Amazon Web Services (AWS) bodo.ai Cloudera Confluent PuppyGraph Snowflake Starburst Upsolver

8

👀 Just a reminder for folks looking for a unified platform in the Microsoft data and AI tooling that automatically captures column level lineage at job and query runtime (including the columns used in an ML model to create a predicted value downstream!), Azure Databricks has you covered. 💪

158

2 Comments

Snowflake: Iceberg Dynamic Tables https://lnkd.in/gtCpgHgu

4

I once had an engineer tell me that they essentially didn’t want to consider cost as they were building a solution. I was baffled. Don’t get me wrong, yes, when you’re building, you iterate and aim to improve your solutions cost. But from my perspective, I don’t think completely ignoring costs from day one is a good plan. In fact, many data teams reaching out to our team in the past 6 months have asked about how they can cut their data stack costs. Cost plays a role in all forms of projects, whether you’re building bridges or writing code. How much budget is allocated to build and maintain a solution is important. In the real world, it can change the materials used, the timeline, or the final product’s design. Whereas in the software and data world, it might push other features and decisions that you make. If anything, cost and performance optimization are likely one of the top things I enjoy doing as an engineer. Sure, it’s fun to build new solutions and infrastructure. But it’s often when we are trying to figure out how to run systems more efficiently or cost-effectively that I’ve felt myself solving a real problem. It forces you to consider methods of storing or processing data in more effective ways that are still easy to maintain, which can feel limiting. There are plenty of common issues that drive up data infrastructure costs, but to find them, you likely first need to approach your cost-saving efforts in an organized way. So let’s go over how you can cut your data stack costs in 2024(5)! https://lnkd.in/g9VGcnCq

60

3 Comments

👀 Just a reminder for folks looking for a modern data warehouse in the Microsoft data and AI tooling that supports identity columns for things like surrogate keys, Azure Databricks has you covered. 💪 https://lnkd.in/gii6qtbW

49

3 Comments

Super exciting recent SqlDBM update! ✨ The release of Tx - transformational modeling! SqlDBM is the first ever ultimate data modeling platform, combining Relational and Transformational modeling in one, built for Snowflake! ❄️ Message me if you’d like a demo! 🚀 #datamodeling #transformationalmodeling #dataengineering

16

1 Comment

Collibra + Databricks enable simplified Data Quality for organizations to benefit from out of the box capability. Get your data ready for the AI journey by leveraging this powerful solution.

42

3 Comments

We did some investigation on JDBC connectors to Salesforce and pricing was ridiculous from multiple vendors. I could employ an FTE for what they were asking. In the age of LLMs, why not just use your new coding assistant to create your own API driven connector? A good example of how LLMs could drive the cost of software down and allow you to invest more in people that can do more because of language models. 😀 Direct queries to Salesforce from Exasol:

36

2 Comments

What a fun 1st "short" to do for Datavolo. I'll be creating more to show Datavolo (and the #NiFi under the covers) is so much more than JUST easier to develop than custom coding -- it is totally customizable, highly performant at enterprise scale, data lineage exists at the EVENT (not just file or table) level, has error handling, it's inherently monitorable/explainable, is production ready, and the list goes on!

2

2 Comments

I have Dataiku. Why do I need Databricks? It's the power of milk and cookies! You could have milk without the cookies or cookies without milk, but you'll get a special treat if you combine them. Let me explain. If you're already using Dataiku to help EVERYONE (not just the tech-savvy) use data science, adding Databricks can boost your data capabilities even more. Databricks handles massive amounts of data efficiently and brings everything—data management, AI, and governance—into one Data Intelligence Platform. It becomes easier for your company to use data on a much larger scale and create new, advanced data projects, like Gen AI applications. Dataiku's simplicity and Databricks' power make a great team, helping your company do more with data faster and safer. And what if the opposite were true? You already have Databricks. Why do you need Dataiku? Milk *AND* cookies, people! Chris Harrison writes a great response here: https://lnkd.in/gKM5cKap. Either way you look at it, our combination brings joy to every executive! #everydayAI

265

13 Comments

Amazing! The dbt Snowflake Native App is now available on Snowflake Marketplace! This integration brings dbt Cloud’s powerful capabilities directly into the Snowflake UI, allowing seamless data discovery, enhanced observability, and AI-driven insights. Key features include: dbt Explorer for in-depth data cataloging. Ask dbt for natural language queries. Orchestration Observability to monitor job runs effectively. https://lnkd.in/egF2Rs8z #dbtCloud #Snowflake #DataIntegration #DataEngineering

12

At dbt Labs Coalesce, I talked with lots of data platform leaders, they all said the same thing: managing RBAC is painful. Many people I spoke with use Terraform or Permifrost (or both!) to manage thousands of roles in Snowflake. Here’s what I kept hearing: » Tools like Terraform and Permifrost are "working" – but barely. People are dealing with hour-long run times, drift issues, and the hassle of manually running SQL. » Most teams stick to the same RBAC strategies to scale. Access roles simplify read/write permissions, while role trees help organize functional and user roles. Right now, Permifrost is the only tool making this strategy easier. » Almost every Permifrost user I met mentioned it's missing key resource types. Some fill the gaps with Terraform, others with custom SQL, but everyone’s a little worried the project might be on life support. Data platform leaders run small, highly efficient teams. A team of 5 might serve an organization of 2000+. To survive, you must invest in automation, CI/CD, and re-usable patterns. The leaders I spoke with are interested in using Titan Core to help them scale. We have made Titan Core the best tool for greenfield projects, the next step is to make it simple to migrate from Terraform or Permifrost. 𝗛𝗲𝗿𝗲’𝘀 𝘄𝗵𝗮𝘁 𝘁𝗵𝗶𝘀 𝗺𝗲𝗮𝗻𝘀 𝗳𝗼𝗿 𝗧𝗶𝘁𝗮𝗻 𝗖𝗼𝗿𝗲: 1/ Migration support for Permifrost. We need tools to convert Permifrost config files, we need better abstractions for grants, and we need published benchmarks for Titan vs Permifrost. 2/ Support for Access Roles. Titan should be the best tool to manage access roles in Snowflake. Users should be able to define read & write roles for a database or schema in 1 line of code. 3/ Resource scope flexibility. Titan should support re-usable configs that only target a single database or schema. This would make it easier to split a large Snowflake account into smaller, easy to manage configs. To achieve that, Titan needs to be more flexible with resource scope. These are my priorities through the rest of 2024. -------- It was great to meet so many of you at Coalesce! Thank you for taking the time to share your stories with me. If you're struggling with Snowflake RBAC, I would love to help – leave a comment, shoot me a DM, or email me at [email protected]

81

14 Comments

Next time you're comparing Snowflake and other platforms, make sure you consider one thing that is always forgotten or ignored but is fundamental to the stability of your platform (especially if you're in financial services or other regulated industries): Snowflake is running by default on minimum 3 availability zones within a cloud provider's region (e.g. Azure West Europe). All active. By default. No configuration. No extra cost. The simplified architecture diagram below should illustrate this. This is often forgotten or ignored, but is crucial for several reasons: 1. No configuration - Before Snowflake, I've worked on projects setting up failover/DR between 2 and 3 data centers, and it is very complex. And expensive. With Snowflake, you get this in all your accounts from day 1. 2. High Availability - the Cloud Services layer ensures failed jobs complete if the underlying compute (VM or hardware) from the cloud provider fails, by retrying jobs on another availability zone 3. Rolling upgrades - Snowflake does not require downtime for upgrades, patches, fixes, etc. which means more time where you can deploy new solutions, and being also able to deploy more solutions. 4. Data protection - data is stored across at least 3 different data centers (of course within the same cloud provider region) similar to locally redundant storage provided by the cloud providers. And did I say this was without any configuration? By default. No extra cost.

275

7 Comments