Dipankar Mazumdar, M.Sc 🥑’s Post

View profile for Dipankar Mazumdar, M.Sc 🥑, graphic

Staff Data Engineer Advocate @Onehouse.ai | Apache Hudi, Iceberg Contributor | Author of "Engineering Lakehouses"

Blogged: Interoperability between lakehouse table formats with Apache XTable (Incubating). Scenario: Dremio -> Apache Hudi ❌ Dremio -> Apache Iceberg ✅ There are 2 analytics teams in an organization. Team A uses Hudi to manage some of their most critical, low-latency data pipelines. Team B focuses on ad hoc analysis, BI, and reporting, using Dremio's compute engine with Iceberg. ❌ Challenge: Team B wants to do analytics using data from the 2 different datasets within the organization. To do so, Team B has to use the dataset generated by Team A (stored as a Hudi table) and combine it with their dataset (the Iceberg table) ✅ Solution: Interoperability between the formats with Apache XTable. Detailed reading link in comments. #dataengineering #softwareengineering

  • diagram
Dipankar Mazumdar, M.Sc 🥑

Staff Data Engineer Advocate @Onehouse.ai | Apache Hudi, Iceberg Contributor | Author of "Engineering Lakehouses"

7mo
Dipankar Mazumdar, M.Sc 🥑

Staff Data Engineer Advocate @Onehouse.ai | Apache Hudi, Iceberg Contributor | Author of "Engineering Lakehouses"

7mo

Join me on May 2nd where I will be talking about all of this work. And always a pleasure to work together with Alex Merced 🙌🏻 https://www.dremio.com/subsurface/sessions/

Like
Reply
See more comments

To view or add a comment, sign in

Explore topics