Airflow Summit

🚀 Exciting News from the Airflow Summit! 🚀 We're thrilled to share a video of our colleague, Eloi Codina Torras, who recently delivered an insightful presentation at the Airflow Summit 2024! Eloi introduced how Meteosim leverages Airflow to streamline operational and tactical decisions related to weather and air quality intelligence. With 7,000 DAG Runs per day, we’ve evolved from using a massive Crontab file to integrating Airflow for enhanced efficiency. In his talk, Eloi dove into our biggest challenge: adapting Airflow to run on multiple Slurm-managed HPC clusters using deferrable operators. By utilizing Slurm, a powerful open-source cluster manager, we significantly improved simulation speed and reduced overhead, making it a perfect fit for scientific and high-performance environments. Additionally, Eloi shared our use case of how we’ve simplified DAG creation—allowing our Product Engineers to write just a few lines of code, while keeping all DAGs standardized and stored in a database. 🎥 Watch the full presentation and learn how Meteosim is driving innovation in the environmental sector through smart, scalable solutions! #Airflow #Slurm #WeatherIntelligence #EnvironmentalTech #Meteosim #AirflowSummit2024

Airflow 3.0 will be the biggest Apache Airflow release in the last five years. I am excited to be part of the community working on it. #Airflow #Astronomer #dataorchestration #DataOps

The first recordings of Airflow Summit 2024 have been published to the Apache Airflow Youtube Channel 💻 We will be having weekly releases so stay tuned and get ready to re-watch your favorite talks or catch any that you missed! 🔗 https://lnkd.in/gZys23Sd

🎉 What an incredible Airflow Summit 2024! 🎉 We’ve just wrapped up celebrating 10 years of Apache Airflow, and it was an event to remember! With over 550 attendees, this was one of the strongest turnouts we’ve seen in the history of the Airflow community. The excitement around Airflow 3.0 was electric! From innovative DAG paradigms to roadmap discussions and hearing how companies like Texas Rangers, Uber, Cloudflare, and Ford are leveraging Airflow to power AI and ML workflows, this Summit had something for everyone. 🚀 Missed a session or want to revisit some of the highlights? Don’t worry—all the talks were recorded and will be uploaded to YouTube soon. 📺 Stay tuned! Huge thanks to all the amazing speakers and panelists for making this event such a success. 🙌 On a personal note, I had an absolute blast with my team at Astronomer! From engaging discussions to tackling an Escape Room!, the camaraderie and collaboration made this Summit even more special. 🗝️🔒 And how could I forget the amazing experience of meeting fellow Airflow contributors in person! Every time we gather, it’s such a great reminder of how strong and vibrant this community is. 🤝 A big shoutout to the Airflow community—we’ve never been stronger, and this Summit is proof that the future of Airflow is brighter than ever. 🌟 Here’s to the next 10 years of innovation and collaboration! 🎉 #ApacheAirflow #Airflow3 #AirflowSummit #AirflowSummit2024 #DataEngineering #OpenSource #GenAI

🚀 If there’s one takeaway from Airflow Summit 2024, it’s this: Airflow 3 will let you “Run tasks anywhere, at any time, in any language.” 🚀 We’re entering an exciting new era with Airflow 3, bringing even more flexibility and power to your workflows. The community was buzzing about this at the Summit, and here’s a sneak peek at some of the features: • A modernized UI that makes managing your DAGs easier than ever before 🎨 • Massive contributor growth - Airflow’s community is stronger than ever! 🙌 • Cutting-edge capabilities that enable task execution across different environments and languages 🛠️ Here are some snapshots from the Summit, including the new UI screenshots and our contributor growth. Shoutout to the amazing Apache Airflow community for making the Summit a huge success! #ApacheAirflow #AirflowSummit #AirflowSummit2024 #Airflow3 #DataEngineering #OpenSource #BigData

🎉 Thank you to everyone who participated in the 10th anniversary of Airflow Summit! We hope you enjoyed these last 3 days with us and we look forward to seeing you in future events 👋

Airflow Summit 2024 is coming to an end! 💡 Remember to attend your favorite talks, expand your knowledge, and connect with industry leaders. 👉 Take a look at our last day's schedule: https://lnkd.in/eqHFm3k

🌟 Ready to keep learning and sharing all the improvements around Airflow for the past decade? Don't miss out on the 2nd day of Airflow Summit 2024! 👀 Check out today's schedule: https://lnkd.in/eqHFm3k

Looking forward to my "First Apache Airflow Contribution" workshop tomorrow at Airflow Summit! There are still a few spots left; if you'll be at Summit, register today to take your first steps toward contributing to Airflow: https://lu.ma/hxcc2ct3

Day 1️⃣ of Airflow Summit 2024! Join us for inspiring keynotes, hands-on workshops, and endless networking opportunities. 🗓️ Remember to check the schedule and attend all of your favorite talks: https://lnkd.in/eqHFm3k