Essam Ammar

💰 💵 𝐂𝐥𝐨𝐮𝐝 𝐂𝐨𝐬𝐭 𝐒𝐚𝐯𝐢𝐧𝐠 𝐓𝐢𝐩: 𝐕𝐀𝐂𝐔𝐔𝐌 𝐲𝐨𝐮𝐫 𝐃𝐞𝐥𝐭𝐚 𝐝𝐚𝐭𝐚𝐬𝐞𝐭𝐬. Managing Delta tables with frequent DML operations can be a storage intensive work, especially when cloud providers charge for all stored data, regardless of its relevance. ⚡ But there is a solution and I have tested it out myself.– 𝐃𝐞𝐥𝐭𝐚 𝐭𝐚𝐛𝐥𝐞 𝐯𝐚𝐜𝐮𝐮𝐦𝐢𝐧𝐠. For those unfamiliar, every operation on a delta table spawns a new version, bloating the underlying data files. Enter vacuuming – a solution to reclaim wasted space by removing older, unused data files beyond a specified retention period. Default? Seven days. The result? Astounding!!. Here are the numbers from one of my tests: 📊 Pre-Vacuum Table Size: 212 TB 📉 Post-Vacuum Table Size: 3 TB 💡 Yes, you read that right – a staggering 98% reduction in storage overhead! 🚨 But, like any solution, there are trade-offs. Vacuuming sacrifices the ability to time-travel back beyond the retention period. However, let's be real – how often do we really need to dig into ancient table versions? 📍 To maximize the impact, it's crucial to complement vacuuming with appropriate storage lifecycle policies. Actively managing versioned data, perhaps by archiving or shifting to infrequent storage tiers, ensures ongoing optimization. Above results speak for themselves. Let's optimize, economize, and innovate together! 💡💻 📚 Reference reads: https://lnkd.in/gVWjF4ew #dataengineering #deltalake #storageoptimization #cloudcomputing #databricks #costsavings #optimization #bigdata #cloud #aws #gcp #azure

6

🔧 Mastering the Factory Design Pattern in Apache Spark 🔧 In the world of big data, designing scalable and maintainable applications is crucial. One powerful approach to achieving this is by leveraging design patterns. Today, I want to highlight the Factory Design Pattern and its application in Apache Spark. What is the Factory Design Pattern? The Factory Design Pattern is a creational pattern that provides an interface for creating objects in a superclass but allows subclasses to alter the type of objects that will be created. It promotes loose coupling and enhances code flexibility and scalability. Why Use the Factory Pattern in Spark? 1. Scalability: Easily adapt to new requirements and data sources by encapsulating object creation. 2. Maintainability: Simplify code maintenance by isolating complex creation logic. 3. Reusability: Reuse the factory logic across different parts of your Spark application. Example in Spark: Imagine you need to read data from multiple sources (e.g., CSV, JSON, Parquet) into Spark DataFrames. Using the Factory Pattern, you can create a unified interface to handle this seamlessly. The example code implementing the Factory Design Pattern in Spark can be seen in the image accompanying this post. By employing the Factory Pattern, you can easily extend your application to support new data sources without altering existing code. Conclusion The Factory Design Pattern is a powerful tool for building robust, scalable Spark applications. It enhances code organization and simplifies the integration of new features. Give it a try in your next Spark project! #BigData #ApacheSpark #DesignPatterns #FactoryPattern #DataEngineering #Scala #SoftwareDevelopment

2

🚀 New on the blog! 📈 We’re excited to share our first article on the blog, focusing on optimizing Apache Spark processing. Discover practical tips to enhance your Big Data workflows, including strategies for partition management, data formats, and memory optimization. 📖 Read the full article here: https://lnkd.in/gNseMTue Get started on boosting your Spark performance today! #DataEngineering #ApacheSpark #BigData #Performance #DataOptimization #TechTips

7

𝐒𝐜𝐡𝐞𝐦𝐚 𝐄𝐧𝐟𝐨𝐫𝐜𝐞𝐦𝐞𝐧𝐭 & 𝐄𝐯𝐨𝐥𝐮𝐭𝐢𝐨𝐧 𝐢𝐧 𝐃𝐞𝐥𝐭𝐚 𝐋𝐚𝐤𝐞. 👉 𝐒𝐜𝐡𝐞𝐦𝐚 𝐄𝐧𝐟𝐨𝐫𝐜𝐞𝐦𝐞𝐧𝐭:  - Suitable for scenarios where the source data schema is static. - Ensures that the source data being written into the target table matches the target table definition, thus providing clean data ready for analytics. - Delta Lake uses schema validation on 𝒘𝒓𝒊𝒕𝒆. This means schema compatibility is checked during the insert operation. - If a row's schema doesn't match the target table schema, the data is rejected entirely, and an exception is raised. - The table schema is not changed unless it is intentionally modified. 👉 𝐒𝐜𝐡𝐞𝐦𝐚 𝐄𝐯𝐨𝐥𝐮𝐭𝐢𝐨𝐧: - Suitable for scenarios where the source data schema is dynamic and you intend to change the table's schema automatically. - Allows users to easily change a table's current schema to accommodate data that changes over time. - Activated by adding .option('mergeSchema', 'true'). - Any columns that are in the source DataFrame but not in the target table are automatically added at the end of the target table definition. 👨‍💻 Below references helps you dive deeper on these concepts.  - https://lnkd.in/gWKx2WVD - https://lnkd.in/gsH4U7_w #datalake #deltalake #schema #databricks #deltatables #dataengineering

7

🚀 𝗟𝗲𝘃𝗲𝗹 𝗨𝗽 𝗬𝗼𝘂𝗿 𝗱𝗯𝘁 𝗠𝗼𝗱𝗲𝗹 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻 𝗝𝗼𝘂𝗿𝗻𝗲𝘆! 🚀 Here’s a quick reference list of essential dbt commands to make your dbt projects more efficient and manageable. These are my go-to commands to streamline transformations, keep projects organized, and optimize workflows. 🌟 𝗱𝗯𝘁 𝗖𝗼𝗿𝗲 𝗖𝗼𝗺𝗺𝗮𝗻𝗱𝘀 🔹 dbt init - Initialize a new dbt project 🔹 dbt run - Execute all or selected models 🔹 dbt test - Run tests defined in your models 🔹 dbt docs generate - Generate documentation 🔹 dbt docs serve - Serve docs locally 🔹 dbt seed - Load CSV data into your warehouse 🔹 dbt snapshot - Capture a table snapshot 🔹 dbt compile - Compile SQL without execution 🔹 dbt debug - Check project configuration 🔹 dbt clean - Remove temporary files 🔹 dbt deps - Install or update dependencies 🔹 dbt source freshness - Check source freshness 🔹 dbt source status - View source table status 🔹 dbt run-operation - Execute custom operations ⚙️ 𝗔𝗿𝗴𝘂𝗺𝗲𝗻𝘁𝘀 & 𝗙𝗹𝗮𝗴𝘀 🔸 --select - Specify models to run or test 🔸 --full-refresh - Perform a full refresh 🔸 --target - Set target environment (dev, prod) 🔸 --threads - Define thread count for parallel runs 🔸 --profiles-dir - Locate profiles.yml directory Mastering these commands can take your dbt projects to the next level! Which ones do you find most helpful? Let’s discuss in the comments! 💬 #dbt #datatransformation #dataengineering #dataanalytics #analyticsengineering #ETL #dbtcore #datamodeling #SQL #bigdata dbt Labs

7

Unity Catalog Migration is a critical step in modernizing data management, and leveraging tools like the Unity Catalog Migration Tool (UCX) from Databricks Labs can make all the difference. At Diggibyte Technologies, we have observed how tools like UCX transform the migration process, offering clear workflows, streamlined transitions, and insightful dashboards that simplify complex migrations. The ability to assess environments proactively and ensure a seamless move to Unity Catalog aligns perfectly with the needs of organizations striving for efficient governance and scalability. If you're embarking on your Unity Catalog journey, tools like UCX aren't just options—they're game-changers. Reach out to the experts at Cuelebre AB to learn how to make your migration process smoother and faster while maintaining data integrity.

12

#SQLTip: Enhance Your UNION ALL Queries with Table Source Tracking When using UNION ALL to combine data from multiple tables, add a 'table_source' column to easily identify the origin of each row. For example: SELECT col1, col2, 'Table1' AS table_source FROM Table1 UNION ALL SELECT col1, col2, 'Table2' AS table_source FROM Table2 This simple addition: - Improves data traceability - Simplifies debugging and auditing - Enables source-specific filtering in subsequent queries What other tricks do you use to keep your UNION ALL queries clean and traceable? Share in the comments! #SQL #DataEngineering #DatabaseTips

10

1 kommentar

When working with an unfamiliar JSON string and needing to identify all keys, using a JSON_KEYS extractor can be extremely useful. #bigquery https://lnkd.in/eY3nKTE6

6

Dealing with private #python packages #Databricks can be painful. Let's say we have package mlops-test stored in a private PyPi repository (we assume basic authentication). The following is required to access this package within your Databricks job: ➡ Secret scope (in our case, "mlops") and secret "pypi_token". The secret scope can be KeyVault-backed (recommended) or Databricks-backed. ➡ Init script to specify extra-index-url. ➡ Unity Catalog Volume (in our example, /Volumes/mlops_test/mlops_volumes/init_scripts) where we upload the init script. ➡ Because init scripts can not access secret scopes directly, we need to create an environment variable. See an example of the Databricks Asset Bundle definition, and read our new article for more details: https://lnkd.in/eBEakGg5 We have not seen it documented anywhere, and hope it will save you from your struggles! #mlops

288

43 kommentarer

🔄 Managing Task Dependencies in Apache Airflow: Trigger Rules, Cross-DAG Dependencies, and Sub-DAGs 🔄 Effectively managing task dependencies is crucial for orchestrating complex workflows in Apache Airflow. Here’s a look at three key techniques: 1. Trigger Rules: • Description: Control when a task gets triggered based on the status of its upstream tasks. • Use Case: Ideal for scenarios where tasks need to run conditionally, such as only proceeding if all previous tasks succeed (all_success) or if any task fails (one_failed). • Example: Ensuring cleanup tasks run only if certain tasks fail. 2. Cross-DAG Dependencies: • Description: Manage dependencies between tasks in different DAGs. • Use Case: Useful for large, modular workflows where different DAGs represent different parts of the pipeline. • Example: Triggering a reporting DAG after the successful completion of an ETL DAG using external task sensors or ExternalTaskSensor. 3. Sub-DAGs: • Description: Embed a DAG within another DAG to manage complex workflows hierarchically. • Use Case: Effective for breaking down large workflows into manageable sections while maintaining visibility within the parent DAG. • Example: Using sub-DAGs for repetitive processes like data extraction from multiple sources. Each method offers unique advantages for specific workflow requirements. By leveraging these techniques, you can design robust and scalable workflows in Apache Airflow. 🚀 #DataEngineering #ApacheAirflow #WorkflowAutomation #BigData #DataScience #ETL #DevOps

1

My favorite tools for #MLOps MLOps end-to-end: #Databricks ML pipelines: Metaflow Platform development experience: Outerbounds CI/CD: GitHub Actions Experiment tracking & model registry: MLflow Feature store: Hopsworks ML monitoring: NannyML LLM task optimization: AdalFlow Model serving: fastAPI Governance: Unity Catalog Remember, MLOps is not about the tools, but how you use them. #machinelearning #datascience

948

58 kommentarer

https://lnkd.in/d7Vh8UBN - Building Data Vault on Databricks with PySpark and Delta Live Tables using streaming/batch AutoLoader - a very performant and scalable way of building enterprise DWH in the Lakehouse ecosystem

7

1 kommentar

Latest Releases from Snowflake, Anthropic AI, Databricks, and Mistral AI https://lnkd.in/dP7uU4mG The Latest AI Innovations from Leading Companies Snowflake: Elevating AI Integration with Cortex Snowflake has introduced Snowflake Cortex, which allows users to integrate AI directly within their enterprise data environments. This includes leveraging AI functionalities for tasks such as sentiment analysis and translation using cost-effective language models. Additionally, Snowflake Cortex enables Python developers to build full-stack AI applications within Snowflake’s secure environment. Databricks: Empowering Generative AI with Lakehouse AI Databricks has introduced new generative AI tools as part of its Lakehouse AI suite, including MLflow AI Gateway and MLflow Prompt Tools, which streamline model management and deployment. Databricks has also optimized its Model Serving for lower latency and cost, and introduced Databricks Lakehouse Monitoring for comprehensive visibility into data pipelines. Anthropic AI: Advancing Safe and Reliable AI Anthropic AI focuses on creating safe and interpretable AI systems, enhancing the safety and reliability of AI technologies. Their flagship model, Claude, prioritizes safety protocols and interpretability in handling tasks, addressing ethical and safety challenges in AI deployment. Mistral AI: Pioneering Open and High-Performance AI Models Mistral AI has released the Mixtral 8x7B model, which excels in multiple languages and coding tasks, and supports selective parameter engagement for faster training and inference processes. They have also demonstrated a commitment to open-source development and strategic partnerships to challenge established players in the AI landscape. Conclusion The latest releases from Snowflake, Anthropic AI, Databricks, and Mistral AI showcase continuous innovation in the AI and data sectors, advancing AI technologies’ capabilities and fostering environments prioritizing security, accessibility, and community collaboration. Practical AI Solutions for Your Business Redefine your work processes and stay competitive with the latest releases from these leading AI companies. Identify automation opportunities, define KPIs, select AI solutions, and implement gradually to leverage AI for your advantage. For AI KPI management advice, connect with us at [email protected]. Explore practical AI solutions such as the AI Sales Bot from itinai.com/aisalesbot to automate customer engagement and manage interactions across all customer journey stages. List of Useful Links: AI Lab in Telegram @itinai – free consultation Twitter – @itinaicom #LatestAI #Innovation #SnowflakeCortex #LakehouseAI #AIModels #productmanagement #ai #ainews #llm #ml #startup #innovation #uxproduct #artificialintelligence #machinelearning #technology #ux #datascience #deeplearning #tech #robotics #aimarketing #bigdata #computer...

3

You can pass the #Databricks job context into your #Python task. It is very useful for reproducibility purposes: ➡ Job run ID can be used to tag your MLflow experiment run and retrieve it later. Experiments in MLflow already have tag mlflow.databricks.jobRunID, but that tag contains the task's run ID (which is not the same as the run ID of the job). ➡ If you save predictions into a table and want to know what run generated these predictions, you can store the run ID and job ID in the column of the table. See the full overview of Databricks job contexts: https://lnkd.in/eQJQGdY7. —————————————————— Hi 👋, I am Maria, MLOps Tech lead with 10+ years of experience in Data & AI, and co-founder of Marvelous MLOps. Follow me for insights on #MLOps, Databricks, and more! Stay tuned and hit 🛎️ not to miss anything!

147

9 kommentarer

https://lnkd.in/dcC-kGpu - Dimensional Modelling vs. One Big Table in Databricks Lakehouse - a very interesting article on data engineering and analytics architecture

27