Gable’s Post

Tired of coordinating multiple copies of your data? A headless data architecture separates data storage, management, optimization, and access from the services that write, process, and query it—meaning that you no longer have to coordinate multiple data copies, and that you are free to use whatever processing or query engine is most suitable for your needs—whether that be Trino, Presto, Apache Flink, or Apache Spark. HDAs can also: ▸ encompass multiple data formats (with data streams and tables being the two most common.) ▸ help with regulatory compliance Learn more about building your own HDA in this 2-part blog series from Adam Bellemare! Read part one here ➡️ https://cnfl.io/4eLBmgB

Tired of coordinating multiple copies of your data? A headless data architecture separates data storage, management, optimization, and access from the services that write, process, and query it—meaning that you no longer have to coordinate multiple data copies, and that you are free to use whatever processing or query engine is most suitable for your needs—whether that be Trino, Presto, Apache Flink, or Apache Spark. HDAs can also: ▸ encompass multiple data formats (with data streams and tables being the two most common.) ▸ help with regulatory compliance Learn more about building your own HDA in this 2-part blog series from Adam Bellemare! Read part one here ➡️ https://cnfl.io/4eLBmgB

💻 How do data contracts work? I'm now writing the book's sections on implementing data contracts with open-source tools. Here is a sneak peek! 👇🏽 The workflow for schema validation: 1. A pull request (PR) is created where the code changes the schema of a database. 2. A new branch kicks off CI/CD workflows where a Docker container is generated. 3. The Docker container has a test database that provides table name and schema metadata. 4. This metadata is used to validate if there is a data contract in place, and if so, check the Kafka schema registry for validation. 5. If the validation shows differing schemas from expectations, the CI/CD check fails, and the ability to merge the branch into main is blocked. 📕 The whole point of this chapter is to take data contracts from a theoretical idea to something you can apply today. I aim to create a public repo with tutorials and code to follow along with the book chapter as well. 🤔 What specific aspects of implementing data contracts do you have questions about? I would love to make sure it's covered in the chapter. #data #dataengineering ----- 🎯 Want more content like this on your LinkedIn feed? Then don't forget to click "follow" on my profile!

Tired of coordinating multiple copies of your data? A headless data architecture separates data storage, management, optimization, and access from the services that write, process, and query it—meaning that you no longer have to coordinate multiple data copies, and that you are free to use whatever processing or query engine is most suitable for your needs—whether that be Trino, Presto, Apache Flink, or Apache Spark. HDAs can also: ▸ encompass multiple data formats (with data streams and tables being the two most common.) ▸ help with regulatory compliance Learn more about building your own HDA in this 2-part blog series from Adam Bellemare! Read part one here ➡️ https://cnfl.io/4eLBmgB

Tired of coordinating multiple copies of your data? A headless data architecture separates data storage, management, optimization, and access from the services that write, process, and query it—meaning that you no longer have to coordinate multiple data copies, and that you are free to use whatever processing or query engine is most suitable for your needs—whether that be Trino, Presto, Apache Flink, or Apache Spark. HDAs can also: ▸ encompass multiple data formats (with data streams and tables being the two most common.) ▸ help with regulatory compliance Learn more about building your own HDA in this 2-part blog series from Adam Bellemare! Read part one here ➡️ https://cnfl.io/4eLBmgB

📄 Mastering Schema Evolution & Type Safety with DataForge 📄 We’ve published a new blog exploring two key challenges in data pipeline development: schema evolution and type safety. Based on recent discussions in the data engineering community, schema changes in source data remain one of the primary causes of pipeline failures. In this post, we cover: 🔹 How schema evolution impacts pipelines, especially when adding, removing, or modifying attributes. 🔹 Why SQL, despite being a strongly typed language, often struggles with enforcing type safety. 🔹 How DataForge addresses these issues with compile-time type safety and automated schema evolution strategies. By leveraging these features, data engineers can: ✅ Improve pipeline reliability and reduce debugging time. ✅ Automate handling schema changes across datasets. ✅ Ensure consistent downstream logic across data lakehouse architectures. Read the full blog and see how DataForge can streamline schema management in your pipelines. 👉 https://lnkd.in/gZ4SYSB7 #DataEngineering #SchemaEvolution #TypeSafety #DataPipelines #DataForge

📄 Mastering Schema Evolution & Type Safety with DataForge 📄 We’ve published a new blog exploring two key challenges in data pipeline development: schema evolution and type safety. Based on recent discussions in the data engineering community, schema changes in source data remain one of the primary causes of pipeline failures. In this post, we cover: 🔹 How schema evolution impacts pipelines, especially when adding, removing, or modifying attributes. 🔹 Why SQL, despite being a strongly typed language, often struggles with enforcing type safety. 🔹 How DataForge addresses these issues with compile-time type safety and automated schema evolution strategies. By leveraging these features, data engineers can: ✅ Improve pipeline reliability and reduce debugging time. ✅ Automate handling schema changes across datasets. ✅ Ensure consistent downstream logic across data lakehouse architectures. Read the full blog and see how DataForge can streamline schema management in your pipelines. 👉 https://lnkd.in/gZ4SYSB7 #DataEngineering #SchemaEvolution #TypeSafety #DataPipelines #DataForge