How do you combine data cleaning with data governance?

As a senior manager in machine learning engineering, it's vital to integrate data cleaning and data governance to ensure data integrity, regulatory compliance, and efficient resource utilization. By combining these practices, your team can foster collaboration, make informed data-driven decisions, and enhance data security and privacy. Emphasize continuous improvement to stay ahead of evolving challenges and maintain the organization's reputation for delivering reliable and impactful machine learning solutions.

We should not eliminate data noise until we are ready to use the data: outliers are windows into latent dimensions, while the amount and statistical distribution of noise tells us a lot about the data generating process. What should be eliminated, and no resource spared in this elimination, is noise added by the data collection system and processes. For example, when we measure temperature by a contact thermometer (as opposed to infrared), we need to introduce a delay to stabilize the thermometer temperature. This is the simplest of the examples. Sometimes we have to do more sophisticated things, like model the effect of the measuring system and infer the value of the unmeasured metric.

In data science projects, ethical principles must be carefully integrated into the processes of data cleaning and data governance to ensure responsible and trustworthy data practices. Through integration, organizations can establish robust data security measures and adhere to privacy regulations, thereby mitigating the risk of data breaches and potential legal repercussions. By conducting data quality checks and validations alongside data governance, the accuracy, completeness, and consistency of the data can be greatly improved, promoting more reliable and credible insights. Eliminating data noise and redundancy enhances the efficiency and effectiveness of data analysis and reporting, allowing for better-informed decision-making.

It's elementary with data: Garbage in, garbage out. This is the lowest common denominator and easiest way to be cognizant on a daily basis.

Combining data cleaning with data governance ensures accurate and reliable data. For instance, in a retail company, data governance sets policies for data quality and security, while data cleaning identifies and rectifies errors in customer data, like name inconsistencies and duplicates. By collaborating, the teams maintain standardized data entry rules, automated validation processes, and accurate customer records. This joint effort guarantees high-quality data for informed decision-making and compliance with regulations.

Data cleaning is the act of normalising and structuring data to make its usage safe and clear. Because this is a tedious, time consuming activity, the natural tendency is to want to reduce the scope by curating before cleaning - so naturally the two become entwined, but this also adds more dependencies in the cleanup process. A better way to combine them is to automate the structuring and cleaning with schema inference and evolution engines, and track schema changes to enable tracing back which column was added to the data set when. During loading, alerting those schema changes to stakeholders and producers of the data also enables better curation and governance.

Rightly put, data cleaning and governance are inter dependent tasks. In my experience, data governance rules determines certain parts of data cleaning whther to mask or anonymize certain fields. For example, data governance rules in the Europe are much stricter whereas in the US not as much, so having to develop different data / ML models based on the region. Data Quality Monitoring on a continuous basis is an essential part of Data cleaning to avoid sudden break in deployed models which has happened a couple of times personally, because upstream decisions introduced data changes but downstream dependencies were not taken care of. Garbage In Garbage Out

You must have at least some kind of model ... Without this, it is unlikely that anything will work out :( It is impossible to obtain a useful result by simple statistical methods.

Data cleaning is unavoidable. Existing and emerging AI tools will provide tremendous help in helping detect data quality issues and help fix them. Apart from that, it's best to start simple. No silver bullet exists (yet) to magically figure out the suitable data format you want and clean it for you. It takes hard work. Treat schemas seriously - often, forgetting to set the proper numeric or datetime format can cause many downstream issues. Lastly, it's worth remembering that this takes time and a lot of hard work. Praise engineers who jump in to fix broken data pipelines, implement checks, and keep improving the existing data standards. This hard work deserves to be recognized.

While data cleaning and governance were traditionally tied at the hip, at the scale of modern data usage it's better to decouple human dependencies. Thus, separating the technical aspects from the human aspects becomes key. Automate your data normalisation through schema evolution engines, and involve a human in the loop for curation via alerts. iIn this way, your data will always be structured, and any changes will be caught on loading before propagating them to the reports.

For data cleaning task, it's important to also take care of the below points: Removing Duplicates: Identify and remove any duplicate records to ensure data integrity. Outlier Treatment: Address outliers that might distort your analysis. Decide whether to remove, transform, or keep them based on the context of your project.

Modern data stacks call for modern technologies. There is a gravitation towards automation through processes of schema evolution. The key reason why schema evolution helps is because it automates the automatable, and then tracks schema changes, enabling alerting them to bring a human in the loop for curation. Let's look at some examples of loading with schema evolution: Bigquery's loading engine will automatically structure non-nested json, if allowed, and you can alert schema changes from BigQuery's information shema. Similarly in modern data lakehouses, schema evolution capabilities exist on transforms too. In the destination agnostic space, dlt library, "data load tool" does schema evolution with alerts for most major destinations.

To keep track of the data cleanliness and schema changes, alerting is superior to monitoring. Usually, we can define more or less what our data should look like, and we can check for that and alert any issues. Monitoring for undefined issues can also be helpful, but this takes splitting the data and inspecting itby various dimensions - which is time consuming and unlikely to happen regularly - it's more common that the user of the data will notice an issue before your monitoring dashboards catch it.

Data cleaning and governance should start at ingestion. Most destinations in which you would ingest data have their own way of loading data and sometimes own support for schema inference and evolution. Major technologies do it to some level but none does it in a "complete" way that you can just plug in. In the destination agnostic space, "dlt" the open source data loading library offers declarative loading with schema inference and evolution, enabling you to have easy schemas, track their evolution and alert changes. Of course the normalisation is automated.

connecting data cleaning to data governance creates a comprehensive and structured approach to data management, fostering data trustworthiness and supporting better decision-making within an organization. Let's say there's a large retail company with multiple stores and an extensive customer database. For instance, in this scenario, and in regards to data governence, establishing data standards can play a key role. To prevent similar data issues in the future, the data governance team collaborates with the IT department to define data entry and integration standards. They create guidelines for data validation, address formatting, and duplicate detection to ensure data consistency and accuracy.

Combining data cleaning with data governance involves integrating data quality standards and data cleaning rules into the governance framework. Establishing standardized data cleaning procedures and metadata management ensures transparency and traceability. Data stewards play a crucial role in overseeing data cleaning activities and enforcing governance policies. Additionally, data governance policies should address data privacy and security concerns during the cleaning process. By documenting data cleaning activities and fostering a data-driven culture, organizations can maintain accurate and reliable data, aligning data cleaning efforts with governance objectives.