DQOps’ Post

Data Observability provides an end-to-end monitoring of data and data processing to detect issues early. The popularity of data observability platforms is driven by the volume and variety of data sources that modern data teams have to handle. Configuring data quality checks to test all datasets is no longer possible. Data engineering teams have two options: ⚡ Wait for the users to report a data quality issue. A data analyst may notice something weird. 👍 Take initiative and observe the data to detect anomalies that precede data quality issues. The second option requires setting up a data observability platform. It should be connected to data sources and data pipelines. The data observability platform will monitor (observe) the data and collect historical metrics, such as the row count. If something weird happens, the engineering team will be notified. The attached data observability cheat sheet lists the most popular data quality problems that data observability platforms are designed to detect. Look into the comments for links to open-source data observability platforms that you could use. #dataquality #dataengineering #datagovernance