Paul Cappelluzzo

Big news from Databricks Conference today. Unity Catalog is going Open-Source. I love all the push for OSS and standards in the data space, right on the back of Snowflake Polaris and the Tabluar acquisition. In the evolving landscape of data engineering and analytics, the push towards OSS and standardization is more crucial than ever. Unity Catalog going open source will mean better data governance availability allowing organizations to benefit from improved data management without being locked into proprietary systems. The benefits for users are immense. OSS leads to larger community support, with better-resulting products at lower costs. They can more easily be extended to specific requirements and integrated seamlessly with existing systems, facilitating smoother workflows and better data governance. Moreover, the community-driven nature of open source means that improvements and new features are continuously being developed and shared. Excited to see these outcomes.

22

3 Comments

There is a lot of dynamics in the #Lakehouse stack these days. It is high time that the community comes to an agreement for a common open catalog standard that can replace the increasingly dated #hivemetastore. For instance, Databricks (#DeltaLake ecosystem) open sourced #UnityCatalog last week: https://lnkd.in/d48Qvm44. Now, this catalog here by HANSETAG is an open source implementation of the #Iceberg catalog REST API, which comes with many bells and whistles for #enterprise deployments such as secure credential mapping. These two are not necessarily just disjunct alternatives when you take into account #Databricks acquisition of Tabular (now part of Databricks) and the plans they shared last week for convergence of #DeltaLake and #Iceberg: https://lnkd.in/dSC2zEf7 At the end of the day people will want a catalog that supports all established table formats and comes with the critical enterprise bells and whistles!

22

1 Comment

Why waste effort for competition wars if you can "join" efforts? I'm excited to hear that 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 agreed to acquire 𝗧𝗮𝗯𝘂𝗹𝗮𝗿 - the company founded by the original inventors of the 𝗜𝗰𝗲𝗯𝗲𝗿𝗴 𝗳𝗼𝗿𝗺𝗮𝘁. The 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 was always focused on the Delta format but opened to the Iceberg format last year when they announced 𝗗𝗲𝗹𝘁𝗮 𝗨𝗻𝗶𝗙𝗼𝗿𝗺 (support for generation of Iceberg metadata for parquet file sets shared with Delta). Now, 𝗗𝗲𝗹𝘁𝗮 𝗨𝗻𝗶𝗙𝗼𝗿𝗺 is GA and the acquisition of Tabular can bring all that experience to another level. Good move Databricks! #databricks #lakehouse #apacheiceberg #tabular #deltauniform

7

There are rumors coming out of Snowflake Summit this week that Snowflake’s launching a semantic layer. Standardizing key metrics like revenue is crucial for streamlining analytics and reporting. DBT and Snowflake's semantic layer products will make this concept mainstream. But metrics aren't the only thing companies lack a standard definition for. Companies care about customers (hopefully) but often don't have a consistent customer definition. Consider a B2C company with customer interactions across multiple channels: 1. Anonymous (guest) visits from multiple devices 2. Logged-in users 3. Physical store interactions Are these interactions considered as one customer or multiple? Is there alignment on this definition between different teams? Is product measuring the funnels for the same set of customers that marketing is targeting with email campaigns? Is data science building life-time-value or churn models for the same customers who are receiving marketing emails? Without a consistent customer definition, teams can't have a shared understanding of the customer's journey. This results in a disconnected experience across product and marketing channels. There's a pressing need for a customer 360 semantic layer. That's what we're building with RudderStack Profiles.

132

1 Comment

When industry giant #Snowflake decided to provide external table management and support for Iceberg tables, it served as a tipping point — the dawn of truly open data architectures is no longer a future ideal; it’s today’s reality. Seeing a major cloud giant embrace open table formats like Iceberg is a game-changer. It signals a fundamental shift: we're moving away from the era where proprietary lock-in was standard, to a data landscape that’s truly interoperable and future-proof. It’s not just about adding “support” for Iceberg; it's a signal that the days of closed, siloed data models are numbered. Open architectures like Iceberg bring autonomy to data teams — allowing them to build pipelines that work seamlessly across clouds, with choice in engines and tools. This means faster innovation and greater flexibility, but it also empowers the entire ecosystem by reducing the risk of vendor lock-in. But the real question is, how far will they go? The next step would be to treat external Iceberg catalogs as first-class citizens — providing true write capabilities with the same performance users experience within the Snowflake catalog. If Snowflake takes this step, it will be transformative for organizations striving to operate seamlessly across data environments without sacrificing performance. And they’re not alone in this vision. Other major data platforms are following suit, building deep integrations with data lakes and open table formats like Iceberg, Delta Lake, and Hudi, further solidifying the industry’s shift toward open standards. This as a massive step toward a genuinely open data ecosystem, one where choice isn’t a trade-off but an advantage. Interoperability, flexibility, and performance can coexist without compromise. 💪 #DataEngineering #Snowflake #OpenData #DataInfrastructure

45

3 Comments

Back in March, #Databricks launched #DBRX—a super-advanced state-of-the-art open language model that's truly next-level. This new model marks a major step forward for #OpenSource #LLMs. What’s really cool is that DBRX outperforms nearly all established open source models and even rivals #ClosedSource giants like #GPT4, #Gemini, and #Mistral across various benchmarks. Want to know what makes #DatabricksDBRX so awesome? We've got you covered! Check out the full article below to learn everything you need to know about it. https://lnkd.in/g3w845G7

2

108 Recipies Snowflake Build: Just moments ago (Nov 12th, 12:30 PM CST), SF presented the Snowflake Dynamic Iceberg table using Polaris External CatalogSync." We have 20+ recipes in this e-book. Below is how to build summary tables as new data streaming into the base table. Snowflake

6

Best practice are a good thing. Switching data platforms to the likes of Databricks, Fabric, and Snowflake can be good also. But even better? Making sure that whatever technical decision you make aligns with your business and enables others to positively impact the bottom-line.

21

2 Comments

As part of Databricks' LakeFlow Council, I've had the opportunity to see the work that is happening being the scenes on LakeFlow Connect, Pipelines, and Jobs. From what I see, "Pipelines" is what DLTs should have been, and MUCH more. About 80% of what I'd like to see appears to be getting implemented. Future appears bright for this one thus far! PS: Shared this with Bilal Aslam's permission.

90

10 Comments

🚀 Unlock the Secrets of Efficient Data Management! 🚀 In today’s fast-paced data-driven world, choosing the right approach to structuring your data is critical. Whether you're working with disparate datasets or handling massive tables, understanding Snowflake and Star architectures can be the key to reducing complexity and improving performance. 💡 In our latest video, we break down: 🔹 How to organize your data using Snowflake schemas for resilience and versatility 🔹 The pros and cons of Star schemas and how they simplify analytics 🔹 Expert tips for navigating the challenges of both models 👉 Don't miss out! Watch the full video on our YouTube channel to master these essential data strategies: https://lnkd.in/dfP4etuP #DataManagement #SnowflakeSchema #StarSchema #DataArchitecture #TechTips #Pynest

4

If you are ingesting data from Kafka/Kinesis into Snowflake you should watch this short video by Charles Tan and Shawn Gordon on how you can use DeltaStream to shift left your workloads to reduce cost and latency! #Flink #Kafka #Snowflake #ShiftLeft https://lnkd.in/gnZ45MTa

13

Elevate Your Data Insights with Databricks Dashboards In this latest article from Blue Orange, we delve into the power of data visualization with Databricks. Discover how to create impactful dashboards that transform raw data into actionable insights: 🟠 Visualize Your Data: Learn how to create clear and engaging visualizations to communicate your findings effectively. 🟠 Leverage AI: Explore the use of AI-powered features to automate data analysis and generate insights. 🟠 Share Your Insights: Share your dashboards with your team to foster collaboration and decision-making. Contact Blue Orange Digital today to schedule a consultation. Databricks https://lnkd.in/enDbHVVb #databricks #dataanalytics #datavisualization #dashboard #AI #datadriven

6

If you've ever had to import a CSV/Excel file to a database and you aren't fluent in SQL it can be a tedious process. You've got to create a new table and get all the data types right before importing the file. Row Zero makes it easy. With a few clicks, any big or small data set in a Row Zero workbook can be saved as a new table in your Databricks warehouse. It's now easy for business teams to get their data into the cloud data warehouse.

18

Do you need to validate your Snowpark data pipelines? Check out this sample framework I built.

40

2 Comments

There is a flood of “Data Quality” vendors in the market. The hype is worse than the hype around AI itself. Almost none of the DQ vendors can explain: -The impact of their tools on your company’s bottom-line. -How their tools are better than homegrown solutions. -What exactly data quality is. The latter, in addition to overtly opportunistic consulting firms, is the reason why there are so many of these tools popping up in the market. This was inspired by Hugo Lu’s post today, and Joe Reis 🤓, you probably agree with this to some extent.

74

28 Comments