DQOps’ Post

One of the core purposes of Data Observability is to detect unannounced table schema changes that can affect the data reliability. Even a tiny column schema change may lead to data loss when the target table cannot accommodate typical data or the source table now contains values that require conversion. One small example can make it clear: 👉 The "registered_date" column in a source table was a DATE type. ⚙️ The pipeline was copying the data to a DATE column in the target table. It all worked flawlessly so for because both types matched. ⚡ The source column changes to STRING or VARCHAR. 👍 Most values contain dates in a valid format, and the data transformation code converts data of dates written as texts to a DATE type under the hood. 💥 It works until a value that is not a date appears in the source table. The purpose of monitoring table schema changes is a core use case of data observability. Check out DQOps, my open-source data observability platform. #dataquality #dataengineering #datagovernance