What if data pipelines were not only bespoke glue code? What if there were more building blocks and best practices to build on? See this template for a local data stack - batteries included. I hope this can serve as a building block to speed things up and build with higher quality & confidence. This Local data stack template is based on #oss #data tools like #duckdb #dagster #dbt #rust tooling for code quality and #pixi for effortless dependency handling https://buff.ly/4feldk6 https://buff.ly/4feldk6 Be advised: Even though it is local it can be easily scaled in the clouds of your choice based on #k8s with smart partition handling.
Georg Heiler
3w
Just recently #fivetran made a pricing change and their transformation support via dbt-core no longer free (https://www.reddit.com/r/dataengineering/comments/1gjoejj/fivetran_just_made_dbt_core_not_free/ https://fivetran.com/docs/transformations/troubleshooting) potentially using this template will allow you to save money
solving critical data challenges
3wIf you want to handle even larger data (> memory of 1 machine) see https://georgheiler.com/post/paas-as-implementation-detail how you can extend this template