Muhammad Ali Zia

I submitted the following papers to #apexconn25: - Fighting Bad PL/SQL & SQL with VS Code - The Trivadis PL/SQL & SQL Coding Guidelines are Dead – What Now? Since I feel bad submitting only in German, I tried something new. I submitted the "same" abstract in both English and German.

17

💛 working with TMDL on Direct Lake I tested my personal new highlight feature of this month today with Marcus Wegener and Andrea Hupp - it works on all explicitly created Semantic Models - the "edit mode" has to no report - like in the Web-Service. But the DAX query view feels "more close" - all changes are synched to the published model. No saving needed. 🤯 but the real game changer is to export the Power BI file a project, then I can: - edit everything in TMDL code and sync the changes fast to the published model (but also beeing able to refuse the changes). Deployment never have been so easy 🤩 - as an empty report appears when I open the .pbip file, I am able to test my changes in real visuals. Even when I love the DAX query view... sometimes visuals show me more insights when I investigate some data issues 😉 👾 Doing the Video was also a chance to use the AI Dubbing one more time - it still feels a little strange, but I still like it and its faster doing the recording multiple times. I still sound different in real english tutorials I do from time to time 😅 Looking forward for more news from the European Microsoft Fabric Community Conference ..! 😃 #powerbi #fabric #update

18

2 Kommentare

What is DataOps? DataOps is a practice that brings agile, DevOps-like principles to data management. It emphasizes collaboration, automation, and continuous improvement to enhance the speed and quality of data delivery. Think of it as the bridge that connects data engineering, data science, and business teams. Why DataOps Matters: 🔹 Boosts Data Quality: Automated testing and validation ensure data integrity. 🔹 Accelerates Insights: Streamlined processes mean faster access to actionable insights. 🔹 Enhances Collaboration: Breaking down silos fosters better communication and teamwork. Core Principles of DataOps: Collaboration: Unite cross-functional teams for smoother data workflows. Automation: Implement automated pipelines for efficiency and accuracy. Continuous Feedback: Use real-time monitoring and feedback to refine processes. DataOps isn’t just a trend; it’s the future of data management. As technology evolves, we can expect even more innovative practices and tools to emerge. Ready to transform how you handle data? Embrace DataOps and discover new ways to boost efficiency and gain deeper insights! #DataOps #DataManagement #Agile #DataEngineering

52

One of the most fantastic and highly fault-tolerant DataWarehouse you could think of is BigQuery- I do not doubt the capacity provided by #Azure and #AWS but again, in my little opinion, Google BigQuery is deemed fit for any data warehouse architecture. Box-in with any ingestion- whether batch or stream pipeline...In the recent time, I worked with my team at D-Aggregate technologies to move data from 3 different backend APIs while collecting retail data from North East and North West Nigeria. The only architecture I could think of is Google Cloud Function(trigger python ETL file) ++ Cloud Schedule(a fully managed enterprise cron-job scheduler) ----> Google Cloud Storage(for effective data storage, gs//)-----> Google Big Query(OLAP architecture) -----> Streamlit(for dashboard data application). Unfortunately, we couldn't, Google rejected all dollar cards, well that's the Nigeria situation, had to deal with lots of issues deploying cloud resources. BigQuery stands out and it is a standard platform to run ML using BigQuery SQL. It easily processes data into any BI tool--> PowerBI, Google Looker Studio and Tableau. I hope Metabase provides a connection soonest. #sql #dbt #snowflake #faulttolerant #data #datawarehouse #ml #ai #genai #google #azure #aws #redshift

21

5 Kommentare

#Medallion Architecture: Simplifying Data Lakes Medallion architecture is a layered approach to managing data in data lakes, often used in platforms like Databricks. It organizes data into three layers: - Bronze Layer: Raw, unprocessed data is ingested from various sources. (This data is typically Imported Data from sources) - Silver Layer: Data is cleansed, transformed, and enriched, making it ready for analytics. ( This data is fed into CRM's, MDM Strats etc) - Gold Layer: Fully refined, aggregated data used for reporting, dashboards, and decision-making. (This layer is Production ready data meets business requirements) This approach brings multiple benefits: - Improved Data Quality: Each layer allows for data validation and cleansing, ensuring only accurate data progresses through the system. - Scalable Analytics: By segmenting data into layers, businesses can perform analytics at the level of complexity they need. - Faster Decision-Making: Refined, high-quality data at the Gold layer provides actionable insights for business leaders. #DataArchitecture #MedallionArchitecture #DataLakes #Analytics #BigData InfoDataWorx

10

Join Databricks on November 7th for Data + AI World Tour Munich! Discover how leading companies like Fraport AG, Deichmann SE, Bayer AG, and CARIAD SE (A Volkswagen Group Company) are taking control of their data and building custom AI on the Databricks Data Intelligence Platform. Register now!

7

2 Kommentare

Hey everyone in Azure Data Engineering! Have you started exploring the Fabric Data Factory Experience? I recently discovered the Office 365 Outlook activity in the new Data Factory experience from Fabric. This addition streamlines your workflow by removing the necessity to create a logic app and set up web activity. Microsoft has introduced this new Office 365 Outlook activity specifically for sending emails. Now, if your task involves only sending email notifications for pipeline success or failure, you can directly use this connector without having to go through the process of creating a logic app and setting up web activity. It's a convenient enhancement that simplifies your tasks! Happy Learning 🙂 #data #azure #microsoft #dataengineer #engineer #cloud #fabric #datafactory #Dp600

74

2 Kommentare

Real and challenging problem while working Prerequisite - Airflow Architecture. The Challenge: As our data processing needs grew, our on-premises Airflow setup started to show its limits. So question is how to scale it? Either go and bring new server but procuring new server is itself a complex task and takes time or utilize Kubernetes cluster. Moving entirely to Kubernetes wasn’t an option—at least not right away—because we were already running critical workloads in production. We needed a solution that allowed us to transition gradually without disrupting everything. What's The Solution: We decided to try a hybrid approach. By running some Airflow workers on our existing VMs and others on Kubernetes, we could slowly shift more tasks to the cloud without a complete overhaul. This gave us a chance to test the setup and build confidence in the new setup before committing fully to Kubernetes. And after regressive efforts Here is the results : This hybrid setup turned out to be promising. We got the scalability and flexibility of Kubernetes while still using our existing infrastructure, with minimized disruptions. It’s been a great interim solution—allowing us to move to the cloud step-by-step, without any abrupt changes. Key Learnings: Gradual Transition is Key: A hybrid approach lets you transition at your own pace, which is crucial when you're already running critical systems. Flexibility: Combining on-premises and cloud resources helped us optimize costs and performance. Proof of Concept, Not Permanent: This setup has been perfect for testing and migrating, but it’s not intended as a long-term solution. Do you see such challenges before or if you want to know how we achieved this DM me. Do comment and follow Rahul Jain for real problem insights. #DataEngineering #ApacheAirflow #CloudComputing #Kubernetes #HybridCloud #TechInnovation #DevOps #DataPipeline #DigitalTransformation #CloudMigration #BigData #Scalability #ITInfrastructure #TechLeadership #ApacheSpark #TechWithRahul #EngineeringExcellence

6

4 Kommentare

Data Model for Inmon Data Warehouse In his book, Building The Data Warehouse, Bill Inmon wrote about the data model for a data warehouse. True, the data model is relational in the format of Entity Relationship Diagram (ERD), then middle level (called Data Item Set) and low-level modelling (physical data model). But immediately after that he wrote about normalisation and denormalisation. "Another important physical design technique that is especially relevant to the data warehouse environment is the deliberate introduction of redundant data", said Inmon. He gave an example of "Description" field which is deliberately placed in many tables redundantly. "In doing so, the access of data is more efficent, and the update of data is not optimal." But "in a data warehouse is no concern whatsoever about update." He then explained about "Snapshots" in a data warehouse at great length. And in the following chapter he spent a lot of time talking about "star join", fact and dimension tables. With clear examples on order fact table, product and customer dimensions, etc. So it is clear to me that Inmon data model is relational (ERD, i.e. Entity Relationship) but he clearly advocates that in a data warehouse we need to do denormalisation, and creating fact and dimension tables. So next time people asked you about the data model for Inmon data warehouse, do not forget the star schema part. The fact and dimension part. Don't forget that Bill Inmon wrote that book in 1996. Can you imagine what data warehousing was like 28 years ago?

47

10 Kommentare

Microsoft Fabric⛓️‍💥 Virtual warehouses with cross database querying Data warehousing items in Microsoft Fabric A SQL analytics endpoint is a warehouse that is automatically generated from a Lakehouse in Microsoft Fabric. A customer can transition from the "Lake" view of the Lakehouse (which supports data engineering and Apache Spark) to the "SQL" view of the same Lakehouse. The SQL analytics endpoint is read-only, and data can only be modified through the "Lake" view of the Lakehouse using Spark. ⚡️ Via the SQL analytics endpoint of the Lakehouse, the user has a subset of SQL commands that can define and query data objects but not manipulate the data. You can perform the following actions in the SQL analytics endpoint: ⚡️Query the tables that reference data in your Delta Lake folders in the lake. Create views, inline TVFs, and procedures to encapsulate your semantics and business logic in T-SQL. ⚡️Manage permissions on the objects. In a Microsoft Fabric workspace, a SQL analytics endpoint is labeled "SQL analytics endpoint" under the Type column. Each Lakehouse has an autogenerated SQL analytics endpoint that can be leveraged through familiar SQL tools such as SQL Server Management Studio, Azure Data Studio, the Microsoft Fabric SQL Query Editor. Synapse Data Warehouse ⚡️In a Microsoft Fabric workspace, a Synapse Data Warehouse or Warehouse is labeled as 'Warehouse' under the Type column. A Warehouse supports transactions, DDL, and DML queries. #Fabric #datawherehouse #Msfabric

10

On my blog I have written about a simple Delta merge process in Databricks. I have links on there to the actual Notebook and files I used if you want to follow along as well as a YouTube video I made! So learn with me the concepts of Delta Lake. https://lnkd.in/ebfkHKrF

15

2 Kommentare

Building a Single-node Lakehouse using 'Hudi-rs' - no JVM, Spark [New Blog] The Apache Hudi community recently released "hudi-rs" - a native Rust library for Hudi with Python bindings. This makes it possible to work with #lakehouse platform like Hudi across a range of use cases that typically don’t require distributed processing. For eg, data science & machine learning applications need frequent access to diverse datasets for exploratory data analysis/model training. Direct access to Hudi datasets for these use cases reduces the ‘wait time’ on data stakeholders, and speeds up the time-to-insight process. Traditionally, accessing these tables involves configuring Java, Spark, Hadoop, and other related dependencies. This is what "Hudi-rs" addresses & makes it easy for data consumers to get started. In this introductory blog, I go over - ✅ the internals of "Hudi-rs" with #ApacheArrow under the hood ✅ how to use it with popular single node computes such as DuckDB, Polars, Daft & Apache Datafusion Check out the link in the comments. And reach out if you want to get involved with the project! #dataengineering #softwareengineering

169

16 Kommentare

It was a good end of the week, wrapping up the DBT Fundamentals course. It is a fascinating piece of tech, allowing you to perform ELT, embed tests for your data models and automated documentation all within 1 tool. Kudos to the dbt Labs training team for designing and explaining the whole content in the easiest possible way. #dbt #dbtlearn #analyticsengineering

38

4 Kommentare

Data Lakehouse vs Data Mesh Data Lakehouse is a data management solution like a database , datawarehouse or a data lake. A simple definition of a lakehouse will be the " best of both datawarehouses and data lakes". For technical folks it is data lake plus a table format (apache hudi, apace iceberg or delta lake). Lakehouse can be implemented in various ways and many vendors have done it in their own way. But some common requirements to satisfy a lakehouse architecture are 1. Handle all types of data 2. Faster data discovery & exploration 3. Reduced ETL 4. Reduced data redundancy 5. Metadata management 6. Use open file formats 7. Decoupled storage & compute 8. Cost effective 9. Integrated security & governance controls 10. Handle multiple use cases. (BI, ML) Databricks Lakehouse, Aws redshift spectrum, Azure Synapse analytics, Dremio are some of the well known lakehouse solutions. Data Mesh: Data Mesh on the other hand is an architectural pattern, which is decentralized and having a product mindset to data. And data mesh can be implemented in many ways right from a database, datawarehouse or a data lakekouse. And data mesh is closely related how the teams are aligned in the organization. The core principles of a datamesh are 1. Domain ownership 2. Data as a product 3. Self serve data platform 4. Federated computational governance To convlude, one is a data management solution and other is an architectural pattern. #datamesh #datalakehouse

1

Data lakehouse capabilities of Apache Doris: ❓ Do you know Apache Paimon? ✍ It is a data lake format that innovatively combines the advantages of the #datalake format and the #LSM structure to introduce real-time stream update capability into the data lake architecture. ❓How is this related to Apache Doris? ✍ Apache Doris provides native support for some of the latest features of Paimon: 1️⃣ Supports various types of Paimon Catalogs, including #HiveMetastore and #FileSystem. 2️⃣ Natively supports Primary Key Table Read Optimized introduced in Paimon 0.6. 3️⃣ Natively supports Primary Key Table Deletion Vector introduced in Paimon 0.8. ❓What does this mean? ✍ It means users can enjoy: 1️⃣ Real-time data ingestion into the data lake: Accelerated by Paimon's LSM-Tree model, data ingestion can be finished within minutes. Paimon also supports aggregation, deduplication, and partial column updates, enabling more flexible data flows. 2️⃣ High-performance data processing and analysis: Paimon provides Append Only Table, Read Optimized, and Deletion Vector. These technologies can be integrated with Doris' powerful query engine to achieve fast queries. 🚀 Comparing Apache Doris 2.1.5 and Trino 422, with Primary Key Table Read Optimized (#Paimon 0.8) enabled, on TPC-DS 1000 dataset: On average, Apache Doris is 3~5X the performance of #Trino.

31

8 Kommentare