Michael Davies’ Post

Want to learn (more) about #dataproducts and how Snowflake utilizes and rely on DataOps.live to build, manage and deliver these at scale? And you happen to be at the #DataCloudSummit in San Francisco? Well, look no further and pencil in this session. You will love it! #dataopslive #truedataops #snowflake #dataproducts #dataengineeringatscale

Recap: Week 2 - Workflow orchestration 🚀 Excited to share my journey in Data Engineering Zoomcamp 2024 by DataTalksClub, and it all kicks off in week 2! Goal: 🔍 Create a seamless workflow orchestration using the Mage platform. What I take: ✔️ Basic orchestration ✔️ Mage.ai | Open-source data pipeline tool for transforming and integrating data. ✔️ Crafting an ETL pipeline to process data from API to Postgres. ✔️ Extending the ETL pipeline to upload to Google Cloud Storage - diving into writing partitioned and unpartitioned data to GCS. ✔️ Continuing the workflow from GCS to load data into Google Big Query. ✔️ Parameterizing and Backfilling Pipelines Credit and a big thank you to Matt Palmer 🙌 for introducing the magic of workflow orchestration, especially through the lens of Mage! ✨ Now, it's time to dive into the homework. 📚 #DataEngineering #ETL #WorkflowOrchestration #dezoomcamp

🚀 The Snowflake World Tour lands this Thursday, 10th October at the Excel with breakout sessions & expert insights—don’t miss out! 🎤 Teaming up with Evoke & AWS for a fireside chat in Breakout Room 3 at 2pm, where we’ll break down how a high-profile merger transformed into a springboard for a cutting-edge, AI-powered strategy. 💡 What’s in it for you? ✅ How to scale your data operations with Snowflake & AWS ✅ Real-time analytics to fuel business agility ✅ Proven ways to justify investment in data innovation ✅ Best practices for data governance & quality ✅ Unlocking Snowflake’s latest AI/ML capabilities ⚡Check out the full agenda here: https://okt.to/9xObPZ 🎟️ Register for Snowflake World Tour London: https://okt.to/cfrVPs

🎉 Just completed an exhilarating journey through the Data Engineering Zoom Camp project! 🚀 🔍 Mastered essential tools like GCP, BigQuery, and Data Studio to design and implement robust data pipelines. 💼 Built a dynamic dashboard visualizing sales insights, empowering data-driven decisions. 🙌 Grateful for the support of instructors and fellow learners. 🚀 Excited to apply newfound skills to drive innovation and impact! #DataEngineering #GCP #BigQuery #DataStudio #SalesInsights #DataDrivenDecisions #LearningJourney #DE #ZoomCamp

🚀 Join Us for the APAC "Best of BUILD" Edition! 🌏 📅 December 9-12 | Virtual Event Dive into Snowflake’s latest innovations from the global BUILD Dev Conference! Explore regionally available advancements with hands-on labs, technical sessions, and insights to empower your journey in AI and enterprise data. 🌟 Highlights: ● Build natural language interfaces for. enterprise apps. ● Fine-tune LLMs with private data. ● Deploy and monetize enterprise solutions effectively. --------------------------------- 📖 What You’ll Learn: Day 1 – Monday, Dec 9 Hands-on Labs (90 minutes each): ● Zero to Snowflake ● Zero Security to Snowflake Horizon ● Zero to Streamlit ● Zero to Snowflake Notebooks ● Zero to GenAI Day 2 – Tuesday, Dec 10 Keynote : ● LLM Avalanche with Snowflake Cortex. ●Building a Governed Lakehouse with Apache Polaris. ●Enterprise-Ready pandas for Snowflake. Day 3 – Wednesday, Dec 11 ● A Python Data Engineer’s Snowflake Journey. ● Preparing Data for RAG Chatbots. ● Pipeline Observability Best Practices. ● Accelerating Productivity with. Snowflake DevOps Day 4 – Thursday, Dec 12 ● Working with Snowflake Connectors ● Building Enterprise-Grade AI Apps ● Creating a Cine_Bot in 30 Minutes --------------------------------- 💡 Why Attend? 16 expert-led sessions tailored to all skill levels (Level 100 & 200). Boost your enterprise data skills and AI expertise. Connect with the vibrant Snowflake community across APAC. 🎟️ Don’t miss out! Elevate your skills and stay ahead in the world of AI and data.

🚀 Data Engineering Zoomcamp 2024 Week 2 Highlights! 🚀 - Week 2: Workflow Orchestration Hello LinkedIn community! I'm thrilled to share the standout moments from the second week of Data Engineering Zoomcamp 2024, hosted by DataTalksClub! This week, we delved into workflow orchestration with Mage. 💻 **Week 2 Recap:** This week, we covered the following key topics: - 📯 Introduction to Orchestration: Understanding the fundamentals of workflow orchestration and how to build data pipelines. - 🧙♂️ Introduction to Mage: Getting acquainted with the Mage platform and learning how to utilize it. - 🐘 ETL: API to Postgres: Building a simple ETL pipeline to load data from an API into a Postgres database. - 🤓 ETL: API to GCS: Exploring how to write data locally to a database and then migrate it to the cloud. - 🔍 ETL: GCS to BigQuery: Loading data from Google Cloud Storage to BigQuery, simulating a common data engineering workflow. - 👨💻 Parameterized Execution: Learning to enhance our data pipelines with parameters. - ⏰ Backfill: Understanding the process of backfilling historical data into the pipeline to ensure completeness and accuracy. - 🔔 Triggers: Exploring how to set up triggers to automatically initiate data pipeline processes based on predefined conditions. 💡 **Looking Ahead:** This week, we ventured into the magical world of Mage, gaining insights into how to manage data workflows effectively. I'm eagerly looking forward to the upcoming weeks! 🔍 **Final Thoughts:** - Thank you to DataTalksClub, Alexey Grigorev for orchestrating this exceptional Zoomcamp. - Thanks to Matt Palmer for explaining Mage in a delightful way this week. - Mage was enjoyable to use as my first orchestration tool. I will use it in my future projects. Bootcamp repo: https://lnkd.in/d9-kwDRP - Homeworks repo: https://lnkd.in/dXZUVNS2 #DataEngineering #Zoomcamp #ETL #Mage #DataOrchestration #DataTalksClub #TechJourney 🌐🔧

🚀 evolv Hackathon Highlight - Turning Data to Value with #Snowflake Last week, evolv's hackathon showcased something extraordinary - the transformative power of Snowflake! Our teams built not one, but three innovative solutions that leverage #Snowflake's robust capabilities: 1. AI-Powered Consultant Matching Application - Combining Streamlit and Snowpark, we developed a smart algorithm that matches consultants to projects flawlessly based on skills and availability. 2. Operational Insights Dashboard - Thanks to Snowflake Native App Framework, we crafted a real-time, interactive dashboard for our #SnowflakeUsage. It’s a revolution in insights, and it's available on the Snowflake Marketplace for effortless deployment! 3. Snowflake Data Ingestion Accelerator - We made data ingestion a breeze. With a metadata-driven approach, our accelerator generates dynamic data pipelines for efficient data integration. These developments are more than just technological leaps; they exemplify our commitment to transforming data into tangible value for our clients and us. Explore Our Thinking to delve deeper into how these solutions can elevate your operations. Let's continue to push boundaries and use #TechnologyInnovation to create exceptional experiences. Here's to many more successful hackathons! #DataTransformation #Innovation

Stay ahead with the latest in tech! Our #technoshare Slack channel has some exciting updates for you: Fivetran's new Managed Data Lake Service is here to revolutionize data lake maintenance. Databricks has just made collaboration easier with the General Availability of Databricks Assistant and AI-Generated Comments. Plus, get introduced to Reladiff for performance-enhanced data diffing. Read more about these updates in our newest blog post. Link in bio. #TechNews #DataScience

Getting the chance to partner with best in class GTM Operators is what makes the art of possible, possible. And thus the chance to work with Travis Henry to unpack how he and the broader Snowflake team are modernizing their GTM was as good as it gets. We geeked out on the the need for establishing golden records for the front office, augmenting data, and building strategic partnerships between our two companies. To quote Travis, "Snowflake is a data company. We think the future of GTM is predicated on attention and focus to data. ZoomInfo is one of our most strategic partners in driving this vision and making it a reality." As the saying goes, iron sharpens iron. Appreciate you, Travis, and the ongoing partnership where we continue to push each other to new heights. https://lnkd.in/gFHSBSsN?

Just wrapped up my final project for Data Engineering Zoomcamp 2024, and I'm thrilled to share it with you all! 🎉 Introducing the Lazio Car Sales Performance & Emissions Dashboard 🚗💨. It's a dashboard designed specifically for stakeholders in the Lazio car industry. Check out the dashboard: https://lnkd.in/d6V_v8UW By combining data visualization and user-friendly filters, this dashboard empowers data-driven decision making for sales strategies, marketing initiatives, and potentially eco-conscious efforts within the Lazio car market. The attached diagram indicates how the project was set up: The diagram shows a monthly batch process that can be triggered manually. It starts with CSV data from the Italian Ministry of Infrastructure and Transportation website (Ministero delle Infrastrutture e dei Trasporti) containing data about all vehicles registered in the Lazio Region from 1911 to 2019. This data is converted to Parquet format, transformed and then uploaded to a Google Cloud Platform (GCP) Bucket. After more transformations, it's stored in a BigQuery data warehouse. Inside the warehouse, DBT is used for further transformations. Finally, the cleaned data is used to create a dashboard in Looker Studio. All of this happens within Mage running on a Docker container. GCP resources are set up in advance using Terraform. For further details check the repo: https://lnkd.in/dPP8S_Zj Thanks to DataTalksClub for the great opportunity! . . . #data #engineering #googlecloudplatform #terraform #dbt #mage #docker #pipelines #etl #batchprocessing #looker