Prefect

🚨 MLOps solution looking for a problem 🚨 (please help 😅) Ben Epstein and I are starting a weekly live stream where we build an end-to-end ML Platform *and* Data Platform from scratch. And then we'll do it again. And again. Using different tools every time to help people understand where tools do and don't shine. We'll set up a data lakehouse. Do ELT and model the data. Do EDA and experimentation. Train and deploy. Monitor and update. Real-time. Batch. Async. We'll try to step back and highlight tool-agnostic principles as we go. 📌 So um... currently looking for some non-boring ML use cases to do this on. Any ideas? (I hate this part 😅) --- Currently, we're using sqlmesh by Tobiko, Prefect, and Modal. Soon we'll look at Featureform. I've been PoC-ing ZenML and Metaflow by Outerbounds at work, so super excited to show off those!

We had a great time hosting the DC Data Community Holiday Meetup last night at #TheEmbassy. The DC data community gathered for an evening of discussion on #ControlFlow with Jeremiah Lowin, bringing local data professionals together to connect and share ideas. If you'd like to join us for the next meetup, check out more information here: https://lnkd.in/emVHTi5C

Discover how modern data teams are revolutionizing their pipelines by combining Python orchestration with dbt Labs. Our latest technical deep dive reveals key insights from industry experts Taylor Curran and Sean Williams. The game-changer? Event-driven workflows that replace traditional cron scheduling. Now your dbt transformations can trigger automatically when data lands in S3, creating truly responsive data pipelines. Key findings from our analysis: - Intelligent pipeline management eliminates bottlenecks - even when finance data takes 8.5 hours to process, engineering reports keep flowing - Seamless integration between Python-based AI tools and dbt workflows enables advanced use cases - like analyzing 30,000 Slack messages using OpenAI - Automated post-dbt workflows can drive personalized user communications based on transformed data Ready to modernize your data orchestration? Our detailed technical analysis includes implementation strategies and real-world examples. Read the full article here: https://prefec.tv/3BpKRnn How are you handling orchestration in your data stack? We'd love to hear your experience 👇 #DataEngineering #dbt #DataOrchestration

At Prefect’s January 2025 Summit, learn how leading organizations are: - Transforming workflow monitoring into true operational visibility - Moving from static infrastructure to dynamic, cost-optimized scaling - Building stakeholder trust through reliable, resilient operations Join technical leaders and industry experts to see how they're driving real business impact with their workflows. January 22-23 | Register now: https://prefec.tv/3ONKUfH #PrefectSummit #DataEngineering

I've started exploring Prefect to orchestrate data workflows seamlessly. From scheduling and retries to real-time notifications and caching, Prefect simplifies managing complex pipelines while enhancing collaboration and visibility. I am excited to share my latest blog on how Prefect empowers data teams to deliver faster, reduce failures, and gain deeper insights into their workflows. Check it out! #DataEngineering #WorkflowAutomation #Prefect #Orchestration #Python

In workflow orchestration, it's easy to get caught in the details: Zoom in: ✓ All data pipelines executed ✓ Tasks completed on schedule ✓ Monitoring dashboards glowing green Zoom out and the story changes: - Your pipelines ran perfectly—but produced unreliable data - Tasks completed on time—but at 3x the necessary cost - Everything looks green—but business leaders have lost trust This is why we're gathering technical leaders at Prefect’s January 2025 Summit: Zoom Out to transform how organizations manage mission-critical workflows and deliver real business impact. Register here: https://prefec.tv/3ONKUfH #PrefectSummit #DataEngineering #WorkflowOrchestration

What if your next strategic hire wasn't a person at all, but a space? While many companies are pushing for return-to-office mandates, we're taking a radically different approach at Prefect: treating physical spaces as strategic hires, each with clear objectives and measurable outcomes. Here's how we're reimagining workplace strategy with three distinct experiments: - The Embassy (DC): A community hub focused on external engagement and collaboration - The Bridge (NYC): A dedicated space for customer product councils and direct engagement - The Lab (Half Moon Bay): A creative retreat for focused technical breakthroughs The key? These aren't just offices—they're strategic assets with specific missions, dedicated ambassadors, and measurable outcomes. And most importantly: team members who don't visit these spaces are held in the same regard as those who do. Read our full perspective on why we're betting on a Remote First future, and how we're making it work below. Our first meetup at The Embassy in DC is happening on Wednesday (12/11). We'd love to have you join us! Check out a link for that meetup below as well.

Today's the day! If you're in Los Angeles, be sure to join us and our friends from Fivetran, Prefect, Sigma Computing, and Dagster Labs tonight for the next stop on our IMPACT World Tour! 🥂 Swing by for dinner, drinks, and an expert panel featuring Dana Neufeld, Data Product Manager at Fundbox who will share her insights on using trusted data and AI. Register here: https://lnkd.in/dMu4PahU See you tonight! #IMPACT #data #AI #dataobservability #dataevents #dataquality

Jumped back into Prefect today for the first time in like 7 months. The UI has come a long way, and setting up a brand new ECS push worker pool was a breeze. That, plus a little DLT, and I had a fairly robust replication pipeline working in a few hours. The data engineering toolset ecosystem is so fun to work with.

Ever wonder why some data pipelines fail to deliver real-time results? Imagine trying to get Taylor Swift concert tickets with a system that only updates hourly! Many organizations are stuck with scheduled data pipelines when they actually need event-driven solutions. Our analysis reveals two crucial approaches: - Pull Architecture: Perfect for efficiency - like monitoring total Snowflake queries without wasting resources on nights/weekends - Push Architecture: Essential for critical events - like catching dropped tables before they cause system failures The key insight? It's not about choosing one over the other - it's about knowing when to use each. Your data pipeline strategy should be as dynamic as your business needs. Want to learn exactly how to implement the right architecture for your use case? Check out our detailed guide that breaks down the pros, cons, and implementation strategies. Read the full article below. What architecture are you currently using for your data pipelines? Share your experiences below 👇 #DataEngineering #Pipeline #DataArchitecture #TechInfrastructure #DataOps