DQOps

📢 Introducing DQOps Data Quality Operations Center – The Open-Source Solution to Simplify Data Quality Frustrated by complex, code-heavy data quality processes? Discover the power of DQOps Data Quality Operations Center. In this quick demo, we'll show you how DQOps delivers a simpler and more intuitive way to keep your data clean, accurate, and reliable. Demo Highlights: - No-Code Data Quality Checks: Intuitively create and run comprehensive data quality checks without writing a single line of code. - Instant Result Verification: Easily review results, pinpoint errors, and identify potential inconsistencies that could impact downstream processes. - Customizable Rule Configuration: Define specific thresholds for warnings, errors, and fatal errors that align perfectly with your unique data requirements. - Data Quality Dashboards: Get a comprehensive view of your data health with customizable KPI dashboards for continuous monitoring. Demo Scenario: Analyzing Data Completeness In this demo, we tackle a common data quality issue: missing values. You'll see how to: - Analyze an essential column and discover that 7.39% of its values are null. - Establish a data quality rule to automatically raise an alert when the percentage of null values in that column surpasses 8%. Key Benefits of the DQOps Open-Source Platform: - User-Friendly: Empower both technical and non-technical teams to manage data quality with ease. - Efficiency Boost: Automate routine checks, eliminate manual errors, and free up valuable time. - Collaborative Ownership: Facilitate data quality discussions across your entire organization. - Cost-Effective Solution: Harness the flexibility and innovation of the open-source model. Ready to transform your data quality? Experience DQOps for yourself! Check out DQOps documentation: https://dqops.com/docs/ #dataquality #DQOps #datamanagement #dataobsevability

Data cleaning is the process of correcting or removing invalid values. There are several groups of data cleaning techniques that you can apply: 👉 Data format standardization methods help to correct small errors without losing information. 👉 Value modification methods covert values to be usable, but at the cost of losing some precision. 👉 Missing values are fixed by data enrichment, such as looking up values from other sources. 👉 Invalid data removal methods focus on removing incorrect records or values. 👉 All other typical data quality errors are handled by detecting errors using data quality checks. Most of these methods can be automated, making it an autonomous process. #dataquality #dataengineering #datagovernance

The difference between data governance policies and standards is simple. Standards are templates, and policies are upper and bottom limits. Data governance standards are ready-to-use templates that can be applied. They meet the best practices in data management and can be activated out of the box in vetted tools. Of course, we may face challenges when applying them to particular datasets. Some tables are too big, others change too often, yet another table is not worth the effort. That is when the data governance policies come to the rescue. A policy should provide the bottom and upper boundaries. 👇 The bottom boundaries are the must-have requirements that all data assets should meet. ☝️ The upper boundaries are the sanity limits to avoid drowning in useless activities. Are there any other activities for which we should have the boundaries set? I could fit only seven areas to make the picture readable. Check the comments for some interesting links. #dataquality #dataengineering #datagovernance

Each data role has its responsibilities but must also communicate with other roles. 👉 Data governance teams create and enforce data rules. 👉 Data architecture teams design the data environment. 👉 Data engineering teams build and maintain data infrastructure. 👉 Data analytics teams analyze data and are closest to the business users. These teams communicate, and we should treat all communication paths as a kind of contract between each group. Should we define "communication contracts" between each team? #dataquality #dataengineering #datagovernance

One of the core purposes of Data Observability is to detect unannounced table schema changes that can affect the data reliability. Even a tiny column schema change may lead to data loss when the target table cannot accommodate typical data or the source table now contains values that require conversion. One small example can make it clear: 👉 The "registered_date" column in a source table was a DATE type. ⚙️ The pipeline was copying the data to a DATE column in the target table. It all worked flawlessly so for because both types matched. ⚡ The source column changes to STRING or VARCHAR. 👍 Most values contain dates in a valid format, and the data transformation code converts data of dates written as texts to a DATE type under the hood. 💥 It works until a value that is not a date appears in the source table. The purpose of monitoring table schema changes is a core use case of data observability. Check out DQOps, my open-source data observability platform. #dataquality #dataengineering #datagovernance

We are talking about the importance of data quality but not how to improve it. It is simple: assess, improve, and prevent. The purpose of any improvement is to find the root cause of problems, fix them and their consequences, and apply techniques to prevent more issues in the future. Data quality improvement requires the same approach: 👉 Assess data quality to confirm the most severe issues. You need to convince business sponsors to go forward with the improvement. 👉 Research the root cause, fix the original problem, and clean up incorrect data. When you are done, pick the next most critical problem to fix. 👉 Prevent future errors by monitoring data quality, retesting all the data quality checks already configured, and reviewing new errors. You can download my free eBook, "A step-by-step guide to improve data quality," which describes this process in detail. #dataquality #dataengineering #datagovernance

There are two ways to ensure healthy data: find and fix them with data quality methods or prevent them by data integrity constraints. At first glance, you could ask why we waste time on data quality management if we could enforce data integrity by database constraints. Well, we cannot verify everything with database constraints. The expressions that we can use are limited and cannot reference third-party data sources. Additionally, if there is no way to force a person to correct a bad record before saving, the invalid records would be ignored and lost forever. It often happens when we receive a copy of data, such as an old list of products or customers. We have to accept the data as it is and find and cleanse the issues - that is the purpose of data quality management. #dataquality #dataengineering #datagovernance

Data Observability provides an end-to-end monitoring of data and data processing to detect issues early. The popularity of data observability platforms is driven by the volume and variety of data sources that modern data teams have to handle. Configuring data quality checks to test all datasets is no longer possible. Data engineering teams have two options: ⚡ Wait for the users to report a data quality issue. A data analyst may notice something weird. 👍 Take initiative and observe the data to detect anomalies that precede data quality issues. The second option requires setting up a data observability platform. It should be connected to data sources and data pipelines. The data observability platform will monitor (observe) the data and collect historical metrics, such as the row count. If something weird happens, the engineering team will be notified. The attached data observability cheat sheet lists the most popular data quality problems that data observability platforms are designed to detect. Look into the comments for links to open-source data observability platforms that you could use. #dataquality #dataengineering #datagovernance

Fixing data quality issues without seeing sample errors is ineffective. When you are cleaning a dataset record by record using a master data management platform, you will see every incorrect record, and you will manually fix it. The problem emerges when reviewing hundreds of potential data quality issues across many tables. We are all discussing the importance of using AI to detect data quality issues. We are also making fun of mistakes that AI makes. Engineers using GenAI must completely rewrite 80% of the generated code. The same happens when we apply automation to data quality testing. Incorrectly configured data quality rules will detect issues that are not valid problems. These issues must be reviewed. If we confirm that some issues are valid but fixing the record-by-record is almost impossible, we must engage the data platform owners. An owner of a business application must designate a budget to add missing data validation at data entry. They need proof that the data quality issue is valid. That is the purpose of data quality error sampling. Pick a few records that did not pass a data quality validation, review them, or download a CSV file you can send to a business sponsor. Check out how the error sampling screen works in DQOps, an open-source data quality platform that you can use locally. #dataquality #dataengineering #datagovernance

Reliable file ingestion depends on validating the schema, constraints, and data quality rules. File ingestion is a process that has been known since most of us were born. Anyway, it is still worth showing the stages where data quality validation should be performed. There are three steps where flat files, such as CSV, JSON, or XML, should be validated: ⚡ Verify the schema and the ability to read the file after copying it to your own raw file location. The file could be truncated (partially uploaded), or it may not have the required columns. ⚡ Use data quality checks that validate constraints, such as value uniqueness, nulls in required columns, and values not convertible to their target data types. You will detect and reject files that cannot be loaded to a typed staging table. ⚡ Run additional data quality checks defined by data stewards and business users. If some of these checks fail, but it is not a critical severity issue, load the data to the target table anyway. This process requires one more important component. You need a job orchestrator that supports restarting jobs at the step where they failed. You can still retry failed jobs if your data transformation code is wrong or the data quality checks are too restrictive. For data quality checks, use a data quality platform that is callable from the data pipelines, such as DQOps. Check my profile to learn more. #dataquality #dataengineering #datagovernance