Data Cafe’s Post

Countdown to Databricks #DAIS2024 is on! If you plan on joining, attend my session! I’ll be providing an overview of the AccuWeather Data Suite and walk through a demonstration highlighting the utilization of our #weather data and #mlflow! See you there.

Imagine data as a treasure—precious, valuable, but sometimes messy. But what happens when the data isn't trustworthy? Enter Delta Lake, the magical solution that brings reliability and consistency to your data. Discover how Delta Lake unifies your data world, ensuring seamless interoperability across Microsoft Fabric. Dive into the magic and maintain data integrity with Delta Lake! For more details, checkout our latest blog on delta lake: https://lnkd.in/d6iR7MTq #DataReliability #DeltaLake #DataEngineering #DataAnalytics #MicrosoftFabric #BigData #Numlytics #DataIntegrity

Is there anyone who worked with a Data Lakehouse? If so, what tech did you use? What is your feedback? In my understanding it makes sense in terms of cost and decoupling the storage & compute. From what I hear Iceberg and Delta lake are common table formats. I'd love to hear your opinions. #dataengineering #data

Simplifying Delta Table Internal File Structure: Ever wondered how Delta Tables manage data so efficiently? Check out this simple visual breakdown of their internal file structure and discover the secrets behind their performance and reliability. #DataEngineering #BigData #DeltaLake

suggested : read Delta Lake basics before this --> https://lnkd.in/eHjwsXuU 💡 Small file problem resolution by bin-packing/compaction #bigdata #deltalake #optimization #databricks #databrickslearning

The video mentioned below is from 2022. In 2022, I was also not a fan of Data Mesh. But I changed my idea with Microsoft Fabric. However, one details I always mention is to have a lot of attention on the original documents, because there are misinterpretations on the video. One I got on the first 5 minutes was that Data Engineers would not be involving in the data intelligence process, the data would be owned by the production developers who created it. That's not what the original Data Mesh papers say. Data Mesh doesn't mean the data producers, the software developers from production, will be the responsible by the product. In fact, the original documents talk about expanding the original team to include data professionals capable to make the data intelligence part of the work. Spreading data engineers among the company may be difficult, it's a fact. But this doesn't mean the responsibility should go to developers. The reason Data Mesh proposes to expand the microservices team to include data intelligence processing is because the business knowledge. The original team, who produced the data, has the business knowledge about it, so the data intelligence work should be done close to them, instead of trying to centralize this knowledge in a central data engineering team. I faced situations like this, centralize the business knowledge in a big corporation may be impossible. Saying Data Mesh is about not having data engineers any more is going with the hype, a lot of hype. From the original Data Mesh paper (https://lnkd.in/eUJfbEN7) : "Domains that provide data as products need to be augmented with new skill sets: (a) the data product owner and (b) data engineers." and "In order to build and operate the internal data pipelines of the domains, teams must include data engineers." The domains teams include data engineers. They will provide the data they manage as a service to other data engineers. Data Product Owners are the ones who knows the business and measures the quality of the data. Data Mesh is not for everyone, but we need to take decisions based on the original concepts, not the hypes. By the way, the sessions I deliver about Data Mesh with Fabric are very well received. I will be delivering one during the Power BI and Fabric Summit and for the Italians, in Pordenone Data Saturday, one train trip of distance from most.