Moataz Ezzo

𝐇𝐞𝐫𝐞 𝐚𝐫𝐞 𝐬𝐨𝐦𝐞 𝐤𝐞𝐲 𝐏𝐨𝐰𝐞𝐫 𝐁𝐈 𝐮𝐩𝐝𝐚𝐭𝐞𝐬 𝐟𝐫𝐨𝐦 𝟐𝟎𝟐𝟏 𝐭𝐨 𝟐𝟎𝟐𝟒! 𝙋𝙤𝙬𝙚𝙧 𝘽𝙄 𝙐𝙥𝙙𝙖𝙩𝙚𝙨 𝙞𝙣 2021: 1️⃣ Premium Per User (PPU) Licensing: Power BI introduced Premium Per User (PPU) licensing, allowing individuals to access premium features at a lower cost. 2️⃣ DirectQuery for Power BI Datasets & Azure Analysis Services: This enabled live connections to Power BI datasets and Azure Analysis Services, enhancing data model flexibility. 3️⃣ Paginated Reports in Power BI Pro: Paginated reports became available to Power BI Pro users, previously exclusive to Premium users. Small Multiples: A new visual feature that enables the display of multiple instances of a visual for various values in a category. 4️⃣ New Model View: Power BI updated the model view interface with diagram layouts, customized properties, and enhanced relationship handling. 𝙋𝙤𝙬𝙚𝙧 𝘽𝙄 𝙐𝙥𝙙𝙖𝙩𝙚𝙨 𝙞𝙣 2022: 1️⃣ Field Parameters: Field parameters allowed users to dynamically select fields or metrics in visuals, simplifying the creation of interactive reports. 3️⃣ Metrics (previously Goals): Power BI Goals evolved into Metrics, offering improved goal-setting and tracking features integrated with Power BI dashboards. 4️⃣ Dataflows Enhancements: Improved performance for dataflows, including DirectQuery support for Datamart and easier management of large datasets. 5️⃣ Error Bars for Line Charts: Error bars were added for line charts, making it easier to display statistical data uncertainties. 6️⃣ Sensitivity Labels for Data Loss Prevention (DLP): Improved security features with the ability to apply sensitivity labels to reports and datasets for better governance. 𝙋𝙤𝙬𝙚𝙧 𝘽𝙄 𝙐𝙥𝙙𝙖𝙩𝙚𝙨 𝙞𝙣 2023: 1️⃣ On-Object Interaction: An intuitive interface to make adjustments directly on visuals, reducing clicks needed to configure reports. 2️⃣ Data Model Improvements: Composite models on Power BI Datasets and Analysis Services allowed more flexible data modeling. 3️⃣ Dynamic M Query Parameters: Allowed the use of parameters to control queries directly in Power BI models, enhancing real-time filtering capabilities. 4️⃣ Data Connectivity Updates: Integration with more data sources, including Google Analytics 4 and Adobe Analytics. 𝙋𝙤𝙬𝙚𝙧 𝘽𝙄 𝙐𝙥𝙙𝙖𝙩𝙚𝙨 𝙞𝙣 2024: 1️⃣ AI-Powered Visuals: Enhanced AI capabilities such as smart narratives and improved AI-driven visuals like Key Influencers and Decomposition Trees. 2️⃣ Expanded Support for Large Datasets: Further improvements for handling large datasets across Premium and Embedded capacities, including autoscale for enhanced elasticity. 3️⃣ Fabric Integration: Integration with Microsoft Fabric for streamlined data integration, governance, and AI analytics across the Power Platform. 4️⃣ These updates reflect Power BI’s continuous focus on usability, AI integration, data security, and performance enhancements.

٤٣

٥ تعليق

BI tools + Data = Data Analyst 👾 Azure / AWS services + Data = Data Engineer 👾 Python / R+ Data. = Data Scientist 👾 Tools might be different, Roles might be different but, here, most Important is data. when you are dealing with a good amount of data with good quality , your achievement will be most efficient.

١٦

Build on your existing PL-300 skills, learn advanced DAX skills, and use lakehouses, warehouses, notebooks and pipelines. #data_engineering, #azure, #microsoft_fabric

٣

Top 10 topics you should know as Data engineer. 1- Database ,DataMart, Datawarehouse , Datalake & Datalakehouse 2- OLTP vs OLAP 3- ETL vs ELT 4- ER Diagram 5- FACT & DIMENSION 6- STAR SCHEMA & SNOWFLAKE SCHEMA 7- TOP DOWN vs BOTTOM UP 8- DIMENSION MODELLING , DATA VAULT2.0 & OBT 9- TYPES OF DIMENSION (Role playing, Conformed ,SCD etc) 10- Incremental vs Full load vs Delta (Upsert) Need resource for all these, drop me a message or comment your email address. #datawarehouse #dataengineer #dataengineering #learning

٣٠٧

٨٧ تعليق

𝑨𝒑𝒂𝒄𝒉𝒆 𝑺𝒑𝒂𝒓𝒌 𝑺𝒄𝒆𝒏𝒂𝒓𝒊𝒐 𝑩𝒂𝒔𝒆𝒅 𝑰𝒏𝒕𝒆𝒓𝒗𝒊𝒆𝒘 𝑸𝒖𝒆𝒔𝒕𝒊𝒐𝒏𝒔 𝒂𝒏𝒅 𝑨𝒏𝒔𝒘𝒆𝒓 𝑺𝒆𝒓𝒊𝒆𝒔: 𝐐𝐮𝐞𝐬𝐭𝐢𝐨𝐧 𝟑 (𝐏𝐚𝐫𝐭 - 𝟐) : 𝐃𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐭 𝐓𝐞𝐫𝐦𝐢𝐧𝐨𝐥𝐠𝐢𝐞𝐬 𝐢𝐧 𝐒𝐩𝐚𝐫𝐤. 𝐍𝐞𝐱𝐭 𝟔 𝐃𝐢𝐬𝐜𝐮𝐬𝐬𝐞𝐝. Excited to share my latest YouTube video where I break down some of the most important Apache Spark terminologies every Data Engineer should know: ✨ JVM (Java Virtual Machine): Understanding Spark's execution backbone. ✨ Lazy Evaluation: How Spark delays execution for optimization. ✨ Cache & Persist: Techniques to supercharge performance. ✨ Checkpointing: Building fault tolerance in distributed applications. ✨ Broadcast Variables: Sharing large datasets efficiently across nodes. ✨ Accumulators: Handling parallel aggregation with ease. To make it even better, I’ve included hands-on code examples to bring these concepts to life. Whether you're prepping for interviews or leveling up your Spark expertise, this video is tailored for YOU! 𝐏𝐚𝐫𝐭 𝟏 (𝐃𝐢𝐬𝐜𝐮𝐬𝐬𝐞𝐝 𝐅𝐢𝐫𝐬𝐭 𝟏𝟓 𝐓𝐞𝐫𝐦𝐢𝐧𝐨𝐥𝐨𝐠𝐢𝐞𝐬) : https://lnkd.in/dB4FYBZW 📺 Check it out here: https://lnkd.in/dZ9R7Wen #ApacheSpark #DataEngineering #BigData #YouTube #TechTutorials

٣٣

١ تعليق واحد

DWH ,Data lake and Data lake house DWH (schema on write) ➡️Data being written in DWH to follow a pre - configuard schema بمعني أن شكل الداتا ويرهاوس معروف زي columns -tables -data types -relations ETL process-extarct,transform and load data in DWH ➡️ high cost ➡️optimized for fast querying ➡️the right option for BI Data lake ➡️ you can add any type of data (structured, unstructured even semi structured) ➡️very low cost ➡️Data lake use ELT Extract data -load data on data lake -transform (data mart or data warehouse) ➡️Data lakes for data science Dis advantage ➡️ complexity ما فيش structure of data واضح ➡️Not optimized for fast querying Data lake house (virtual DWH) ➡️single unified storage اقدر اخزن الداتا بتاعتي كلها زي data lake ➡️Query optimized عكس data lake ➡️Flexible schema اقدر اعدل علي الاسكيما ➡️ it's a complex job استخدم ايه ودا عشان اجاوبه لازم اجاوب علي اسئله تانيه ؟؟!! - what are the business goals? - what are the skill set that we employee? - Data types distribution? - Amount of data generated? #ETL #DWH #Database #Data_Engineering #Bi

٤٠

Data Governance: What is #Data_Governance? Data governance refers to the overall management framework that ensures high data quality, availability, usability and security within an organization. So unlock the potential of data governance for effective management and strategic utilization of organizational data assets. Why #Data_Governance? So data governance is required for data quality assurance, risk management and security regulatory compliance, support for decision making and strategic planning, and operational efficiency and consistency.

١

💻 𝐎𝐋𝐓𝐏 𝐯𝐬. 𝐎𝐋𝐀𝐏: 𝐔𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝𝐢𝐧𝐠 𝐭𝐡𝐞 𝐃𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐜𝐞 📊 In the world of data management, understanding the roles of OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing) is essential for designing efficient systems tailored to specific needs. 🔹 𝐎𝐋𝐓𝐏 (𝐎𝐩𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐚𝐥 𝐃𝐚𝐭𝐚) is designed to handle day-to-day transactional tasks:   - Transaction-oriented and detail-focused.   - Operates on entity-relationship (ER) models.   - Manages smaller datasets, ranging from 100 MB to a few GB.   - Fully normalized data structure. 🔹 𝐎𝐋𝐀𝐏 (𝐈𝐧𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧𝐚𝐥 𝐃𝐚𝐭𝐚) focuses on data analysis for decision-making:   - Object and market-oriented, suitable for analysis.   - Manages vast historical datasets using star or snowflake models.   - Supports large data volumes (100 GB to TB).   - Partially normalized structure, operating on millions of files. Choosing the right approach depends on your needs: - 𝐎𝐋𝐓𝐏 for real-time transactions, speed, and detail. - 𝐎𝐋𝐀𝐏 for deep analytics, trends, and historical insights. 🌐 we leverage both to streamline operations and drive insights! Which one does your organization rely on the most? #DataManagement #Analytics #OLTP #OLAP #BigData #DigitalTransformation

٤٨

## My Azure Data Factory Journey: A few weeks ago, I decided to learn Azure Data Factory (ADF), and it's been a transformative experience! Although I haven't integrated it into my workflow yet, the learning process has been eye-opening. Having used SQL Server Management Studio (SSMS) and SQL Server Integration Services (SSIS) , I wanted to share some key insights and advantages I've found with ADF. Real-Life Benefits Over SSMS and SSIS 1. Cloud Power   - Scalability: ADF scales effortlessly with your needs, no more worrying about server limits   -Managed Service: Less maintenance, more focus on actual data tasks 2. Cost Efficiency   -Pay-As-You-Go: Only pay for what you use, which is much more economical than maintaining on-prem infrastructure 3. Integration Made Easy   - Wide Range of Connectors: Easily connect to both cloud and on-premises data sources   - Hybrid Integration: Seamlessly combine data from different environments 4. Advanced Features   - Data Flows: Visual, code-free data transformations   - Pipeline Orchestration: Intuitive drag-and-drop interface with excellent monitoring tools 5.Security and Compliance   - Built-In Security: Robust security features like data encryption and Azure Active Directory integration ### Final Thoughts Learning ADF has been a game-changer. The process has not only expanded my skill set but also opened up new possibilities for future projects. If you're thinking about learning ADF or just getting started, dive in. It is worth it. #AzureDataFactory #DataIntegration #CloudComputing #TechJourney #DataTransformation CapItAll

١٣

🔄 **Master Recursive CTEs for Advanced Data Queries!** 🔄 In SQL Server, Recursive Common Table Expressions (CTEs) are a powerful tool for handling hierarchical data and performing complex recursive queries. Let's dive into what they are, their benefits, potential drawbacks, and practical use cases. ### What is a Recursive CTE? A Recursive CTE is a CTE that references itself, allowing you to perform recursive operations. It's particularly useful for querying hierarchical data, such as organizational structures, folder hierarchies, or graph data. ### Why Use Recursive CTEs? - **Simplicity**: Simplifies complex hierarchical queries that would otherwise require intricate self-joins or iterative procedures. - **Readability**: Enhances the readability and maintainability of recursive queries. - **Flexibility**: Handles variable-depth hierarchies efficiently. ### Example Scenario Suppose you have an organizational structure and you want to retrieve the entire hierarchy of employees under a specific manager. ### Sample Code ```sql WITH EmployeeCTE (EmployeeID, EmployeeName, ManagerID, Level) AS ( -- Anchor member: select the top-level manager SELECT EmployeeID, EmployeeName, ManagerID, 0 AS Level FROM Employees WHERE ManagerID IS NULL UNION ALL -- Recursive member: select employees reporting to managers in the previous level SELECT e.EmployeeID, e.EmployeeName, e.ManagerID, c.Level + 1 FROM Employees e INNER JOIN EmployeeCTE c ON e.ManagerID = c.EmployeeID ) SELECT EmployeeID, EmployeeName, ManagerID, Level FROM EmployeeCTE ORDER BY Level, EmployeeName; ``` ### Use Cases 1. **Organizational Hierarchies**: Query hierarchical employee structures, reporting lines, and departmental relationships. 2. **Bill of Materials (BOM)**: Manage and query multi-level BOMs in manufacturing. 3. **Folder Structures**: Query nested folder hierarchies in file systems or content management systems. ### Advantages - **Conciseness**: Reduces complex iterative logic to a few lines of code. - **Efficiency**: Handles deep hierarchies efficiently without the need for multiple joins. - **Maintainability**: Makes recursive logic easier to understand and maintain. ### Disadvantages - **Performance**: Can be less performant for very deep or broad hierarchies. Optimization and indexing are crucial. - **Complexity Limits**: SQL Server has a default recursion limit (100 levels). Deep recursions might require adjustments or alternative approaches. - **Debugging**: Recursive CTEs can be challenging to debug, especially with complex logic or large datasets. ### Best Practices - **Indexing**: Ensure proper indexing on columns used in recursive joins to optimize performance. - **Limits**: Use the `OPTION (MAXRECURSION n)` hint to control recursion depth and prevent runaway queries. #SQLServer #Database #DataAnalytics #RecursiveCTE #DataScience #DataManagement

٥

📙 DWH vs Data lake vs delta lake  =================================== DWH : Only Structured data Data Lake : Structured/Semi-Structured/Unstructured  Delta Lake : Structured/Semi-Structured/Streaming DWH:Schema on write Data Lake :Schema on read Delta Lake :Schema on read DWH:Supports ACID Transaction Data Lake : Minimal support to ACID Transcation Delta Lake : Support ACID Transaction DWH: Does not corrupt the system Data Lake : Leaves system in corrupted state Delta Lake : Does not corrupt the system #databricks #spark #dataengineer #deltalake 👉 Follow Aman Maheshwari for more such content.

٤٤

BI / Data strategy ----------------------------- Offering : Domains data products on SQL Lakehouses and DWH ( thanks to DBT) and analytics cubes stacks ( power bi , tableau , looker ..) on strong data quality under strong governance policy ( Big skills , persona , domains ( SQL + analytics together ) owners , sharing , security , rules , tests , cicd ..) with data objects well documented via a data catalog service...is the key of success The way for a silver or gold medal 😜 Most of the time my first as fireman actions target to STOP hundreds of power bi semantic models duplicates ( due to 5 last years a not governed self bi policy)

١

#Data_governance Data lineage and traceability Problem A manufacturing company struggles to trace the root cause of a product defect due to unclear data lineage. Solution Data governance provides clear documentation of data origins, transformations, and flows, enabling the company to trace the defect’s source and implement corrective actions.

٨

As data engineers, one of our key responsibilities is to manage data efficiently and cost-effectively. Here are some valuable tips to minimize storage costs while implementing batch datasets in Parquet format using Azure technologies: 𝗞𝗲𝘆 𝗧𝗶𝗽𝘀: 𝗖𝗵𝗼𝗼𝘀𝗲 𝗣𝗮𝗿𝗾𝘂𝗲𝘁 𝗙𝗼𝗿𝗺𝗮𝘁: Parquet is a columnar storage file format that is highly efficient for both data storage and retrieval, particularly for analytical queries. It reduces storage costs by offering better compression and faster query performance. 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗲 𝗗𝗮𝘁𝗮 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝗶𝗻𝗴: Effective partitioning of your data can significantly reduce the amount of data scanned during queries, leading to cost savings. Partition your data based on high cardinality columns, such as date or category. 𝗟𝗲𝘃𝗲𝗿𝗮𝗴𝗲 𝗖𝗼𝗺𝗽𝗿𝗲𝘀𝘀𝗶𝗼𝗻: Parquet supports multiple compression algorithms. Use efficient compression methods like Snappy, which offers a good balance between compression ratio and speed, thereby reducing storage costs and improving performance. 𝗨𝘀𝗲 𝗗𝗲𝗹𝘁𝗮 𝗟𝗮𝗸𝗲: Delta Lake on Azure Data Lake Storage Gen2 can further optimize storage and transaction management. It provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing, reducing the need for data duplication and thus storage costs. 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗲 𝗙𝗶𝗹𝗲 𝗦𝗶𝘇𝗲𝘀: Large numbers of small files can lead to higher storage and processing costs. Aim to optimize the file size by adjusting the number of records per file to balance between processing speed and storage efficiency. 𝗠𝗼𝗻𝗶𝘁𝗼𝗿 𝗮𝗻𝗱 𝗠𝗮𝗻𝗮𝗴𝗲 𝗦𝘁𝗼𝗿𝗮𝗴𝗲: Regularly monitor your storage usage with Azure Cost Management and make adjustments as needed. Consider lifecycle policies to automatically move older data to cooler storage tiers to save costs. By following these tips, you can not only minimize storage costs but also enhance the efficiency and performance of your data pipelines. Feel free to share your own tips and experiences in the comments below! Let's empower our data engineering community together. #DataEngineering #Azure #BigData #Parquet #CloudComputing #DataLake #AzureSynapse #CostOptimization

٧

٢ تعليق

ETL Project with SSIS & Power BI ✨ 📍Project Overview: ✅ This project entails an ETL (Extract, Transform, Load) process utilizing SSIS, followed by data visualization through Power BI. The integration of SSIS facilitates seamless extraction, transformation, and loading of data, ensuring its accuracy and efficiency. Subsequently, Power BI is employed to present the processed data in a visually appealing and insightful manner, empowering stakeholders with actionable insights and facilitating informed decision-making processes. 📍Steps: ➡️ Extracting data from the CSV file. ➡️ Performing data conversion. ➡️ Placing the data into the source database (Customers table) for SQL query execution. ➡️ Transferring the data from the source database to the Data Warehouse and creating dimension tables and a fact table. ➡️ Loading the data into the dimension and fact tables. ➡️ Generating insights using SQL and converting them into views for utilization in Power BI. ➡️ Finally, creating reports using Power BI. If you want to learn SSIS you can learn it from Eng. abdelrahman mamdouh 🇵🇸 : https://lnkd.in/dV3XAmaK Thanks for your time❤

١٦٩

١٦ تعليق

#Opportunity for #dataanalysts #QA #Developers to upskill at next level to work with #realtime #scenarios. Here is a small project which will improve your skills and we do provide such upskilling learning everyday. Follow - Yogesh Tyagi #data #sql #dataanalysis #datascinece https://lnkd.in/grRR5vAP

١٦٧

٨٩ تعليق

🚀 Understanding Apache Spark Architecture! 🌟 #Spark_Application_Submission: When you submit a Spark application, it is packaged and sent to the Spark Driver. #Spark_Driver: The Driver is responsible for orchestrating the application. It handles the execution plan and maintains information about the Spark jobs. #Application_Master: The Application Master negotiates resources with the Cluster Manager and manages the execution of the application. #Cluster_Manager: The Driver communicates with a Cluster Manager (like YARN, Mesos, or Kubernetes) to request resources. The Cluster Manager allocates resources (CPU, memory) based on availability and the needs of the application. #Spark_Executors: Once resources are allocated, the Cluster Manager launches Spark Executors on worker nodes. Executors are the processes that run tasks and manage data storage for your application. #Spark_Session: After the executors are launched, a Spark Session is created, which serves as the entry point to interact with Spark functionality. It encapsulates the context for your application. #Task_Execution: The Driver breaks down the application into smaller tasks and schedules them to run on the Executors. Each Executor runs its assigned tasks and processes data. #Result_Collection: Once the tasks are completed, the Executors send the results back to the Driver. The Driver collects these results, which can then be used for further processing or displayed. #Final_Output: Finally, the Driver presents the output of the application, whether it’s writing to storage or returning data to the user. #ApacheSpark #BigData #DataEngineering #TechInsights #Spark

٢٩

٢ تعليق

DBCC CHECKDB with estimate only in SQL Server

٤٥

١ تعليق واحد