Airflow Summit’s Post

Looking for a way to streamline your data workflows and master the art of orchestration? Airflow Datasets can be easily integrated in your data journey. This session will showcase the Dynamic Workflow orchestration in Airflow with a real-time #datapipeline and dbt in Google Cloud environment. Save your spot: https://airflowsummit.org/ #AirflowSummit #Airflow #ApacheAirflow #AirflowSummit2024 #dataorchestration #batchprocessing #dataprocessing #datapipelines #data #datasience #opensource Apache Airflow

The Power Automate cloud flow run history in Dataverse, you can apply the extensibility of Dataverse to track the results of your cloud flow executions at scale. this feature in Preview, can use the power of Dataverse’s common data architecture, including Role-Based Access Control (RBAC), to manage the Flow Run data. Only solution cloud flows, with their definitions in Dataverse, can have their run history stored in Dataverse. #Dataverse #Microsoft #PowerPlatform #CloudFlow #PowerAutomate

Looking forward to see you in San Francisco. Meeting myself & Nawfel Bacha how we are handling Airflow Dataset in Astrafy data pipeline. #airflow#dataengineering#summit

💡Get expert tips on how to master #AzureDataFactory (ADF) and Apache Airflow! Join our webinar next week to learn how to: 🔹 Identify ideal use cases for combining #ADF and Airflow 🔹 Orchestrate multiple ADF pipelines with Airflow as your single control plane 🔹 Leverage Airflow's flexibility for dynamic ADF #pipelines 🔹 Kickstart #Airflow in production on Microsoft Azure Save your spot 👉 https://lnkd.in/g6YYzFkK #azure #datapipelines #dataorchestration #dataengineering

Watch this demo to see how Bitwise accelerates conversion of SnowSQL code to PySpark for execution in AWS Glue. Learn more about our automated Data and Database Objects cloud migration solutions: https://bit.ly/3VM0CwV #SnowSQL #PySpark #AWSGlue #DataMigration #CloudMigration #DatabaseMigration #Automation

When developing or debugging a stream processing pipeline with Flink SQL, it’s common to inspect each processing step's output to ensure data is being transformed properly. However, comprehending the resulting data stream's structure, distribution, and characteristics entails executing multiple ad-hoc SQL queries, which can be time-consuming and tedious. Additionally, isolating specific subsets of the stream for analysis or debugging often involves even more queries, adding to the complexity and time required. Interactive tables for Flink SQL Workspaces allow you to quickly explore and visualize Flink query results directly within the Confluent Cloud UI. Users can efficiently scan, analyze, and profile the output data of each query, streamlining the development and troubleshooting processes when implementing stream processing workloads. #confluent #apacheflink #flink #streamprocessing #datastreaming https://lnkd.in/daJtiiby

🌟 Excited to share my recent project in data engineering on the cloud! end-to-end cloud-based data pipeline to process high volume data. Here's a quick breakdown of what we achieved: 🔹 Automated data extraction using Apache Airflow and API integrations. 🔹 Leveraged Azure Data Lake Storage Gen2 for scalable and secure data ingestion. 🔹 Performed data transformation with Databricks, Delta Live Tables, and Unity Catalog for governance and efficiency. 🔹 Streamlined orchestration and scheduling with Azure Data Factory into databricks for reliability. This project not only enhanced data processing speeds but also ensured scalability and data integrity across the pipeline. 🚀 Cloud technologies like these are truly redefining how we handle big data! #DataEngineering #CloudComputing #Azure #Databricks #BigData #DataPipeline

🚀 Simplify and Orchestrate Your Workflows with Cloud Composer! ☁️✨ In the era of Big Data and Cloud-Driven Automation, managing workflows has never been easier. Say hello to Google Cloud Composer – the fully managed orchestration tool powered by Apache Airflow. Why Cloud Composer? 🌟 🔹 Seamless Workflow Orchestration: Automate, schedule, and monitor complex workflows effortlessly. 🔹 Fully Managed & Scalable: Focus on your business logic while Google Cloud handles the infrastructure. 🔹 Cross-Cloud Integration: Connect workflows across GCP, AWS, Azure, and on-prem systems. 🔹 Data Pipeline Flexibility: Build end-to-end ETL/ELT processes with robust Apache Airflow capabilities. 🔹 Reliability & Visibility: Monitor workflows with intuitive dashboards and real-time alerting. Use Cases: ✅ Automated Data Pipelines ✅ Batch & Streaming Job Orchestration ✅ Data Warehousing Integrations (BigQuery, Snowflake, etc.) ✅ Machine Learning Workflows ✅ Cross-Service Automation 🔥 Take Control of Your Data Pipelines with Cloud Composer – build workflows that scale and deliver on time, every time. 💬 Are you using Cloud Composer yet? Let’s connect and explore how this powerful tool can orchestrate your workflows and streamline your operations! 🚀 #CloudComposer #GoogleCloud #DataEngineering #ApacheAirflow #WorkflowAutomation #ETL #BigData

Imagine a simple yet powerful storage solution that grows alongside your business, eliminates the guesswork of capacity planning, and offers predictable costs with no surprises.... It’s here! Learn more about Real-time Enterprise File and how it can help make your job easier and your data everlasting: https://purefla.sh/3YJeesq #data #DataStorage #PureStorage #IT #innovation #StoragePlatform #FileAndObject #AI #GenAI #DataCenter #sustainability #STaaS #SaaS

Wildly full room for Ana’s and my “Power Platform in a Modern Data Platform Architecture” session at European Power Platform Conference in Brussels 🇧🇪🤩 👇 White paper on the topic in comments below… #Microsoft #Cloud #AI #AIstrategy #CloudStrategy #Azure #PowerPlatform #msFabric #m365 #dynamics365 #CloudEcosystems #CloudLighthouse #EcosystemShow #EPPC24