How do you update your data quality skills?

ISO 8000 defines quality data as portable data that meets stated requirements. So meeting expectations and requirements is not enough. First and foremost data must be interoperable. If data is only readable by one application it is part of the application (and subject to the application license). Learning what makes data interoperable is a fundamental data quality skill. Take the free ECCMA Master Data Quality Manager course to learn more.

The best way to update data quality skills is essential in today's evolving landscape. Engage in continuous learning through online courses, workshops, and webinars. Practice with real-world datasets to hone skills. Join data quality communities to exchange insights. Stay current with industry trends, tools, and best practices. Regularly read blogs, and research papers, and attend conferences. Incorporate feedback from peers and supervisors. Adaptability and commitment to improvement are key to mastering data quality.

The basics of Data Quality: accuracy, completeness, consistency, validity, uniqueness, and timeliness. It is very important to know where and what to check: when to check null values because you don't expect them and when to check that value is aligned with business logic.

Contrary to popular belief, the most successful data projects starts from the business sponsors in collaboration with technology. In marketing, for example, the guiding drivers of descriptive and predictive analytics, KPIs and dashboards build the domain context for the 6 core dimensions of DQ - completeness, consistency, accuracy, validity, timeliness and uniqueness. If this can be nailed, rest is a matter of creating, managing and executing the DQ rule either custom build, part of tech stack or park of a library of DQ rules. The top DQ professionals have a mix of domain and tech stack knowledge. But to start one can learn with online learning portals of from Informatica , Qualitystage, Collibra, Talend, SAS, Expedian and others

One of simplest trick I use to improve my data quality. To understand company needs and requirements. Than i easily separate the data as per the requirements and do some comman data quality chevk ✔️

Data Quality is a not just a technical way of handling any Data issues, it is more around understanding Business Processes, standardize data formats, units, and naming conventions to improve consistency across all data lake. Train your team on data quality best practices and raise awareness about the importance of data quality across all levels of enterprise to keep a close tab of all the business-critical processes. Assessing, analyzing, and improving data quality is essential for organizations to make informed decisions and avoid potential risks. Regularly revisit your data quality strategy and adjust it as needed to address new challenges and changing business requirements.

During my PhD, I've read all the popular data quality books and I've noticed they all say more or less the same and overly emphasise the importance of concepts like "DQ dimensions" and "fitness for use". One book that stood out to me was "Total Information Risk Management" by Borek, Parlikad, Webb and Woodall. If you want to approach data quality from a pragmatic point of view, you could start there!

Understand the business rules and objectives. The data is only useful when it can serve the business purpose. My opinion is, it is no use adopting world best data quality standards and practices if it cannot meet the business needs or it bogs the business down with unnecessary overheads. I always adopt a pragmatic approach to data quality, finding a balance between quality and business needs and business objectives.

Data must serve the business so your main concern should be, "How do you put a dollar value on a data source?" The answer lies not just in the data itself but in the relationships you maintain with the people and systems that produce it. A harmonious relationship ensures a seamless flow of high-quality data. The age-old adage "garbage in, garbage out" holds especially true for data. If you're working with dirty data, don't expect insightful outcomes. Time also plays a crucial role. Some data elements like addresses, phone numbers, and ages decay over time. Regular updates are non-negotiable. Improving data quality isn't just a one-off task; it's a continuous process that requires change management. Plan your strategy accordingly.

Yes this should be the starting point, where we should understand the data quality process and how critical data quality is for an organisation.

Standards and best practices for Data Quality are good but don't fall into the trap of following and repeating 100% of the guides and standards. Adjust when needed to the size and needs of your company, your storage, and business requirements. Some suggestions from those guides and Frameworks work only on big companies, for example.

Every business is different and there is no single size or process that fits all, so we need to understand what's the pain points what are the sources and then, we need to implement the processes as per best practices from DAMA DMBOK or other data governance councils. It's should be your own process associated with your business complying the standards.

Standards are great, but explaining standards used and why they have been chosen is the hardest part. In many organizations many stakeholders need to be involved in a data quality improvement initiative. This involvement is necessary but not necessarily very easy to manage. Deciding to take their (imagined) data standards up for review and involving stakeholders across different business functions can often be a very challenging. Understanding the WHY and making it a fundamental focus of all communication when involving them is key to success. Selling it as an enhancement of the business data usage across functions and different silos as the biggest gain will be beneficial, creating a positive outcome for a common model for data quality .

Adhering to standards in data quality is crucial for consistency and reliability. Standards like ISO 8000, DAMA DMBOK, and others provide a structured approach, ensuring we're not just "winging it." Each industry and organization has its unique needs, so aligning with specific standards and policies is key to meeting business objectives. It's about integrating data quality into the fabric of organizational processes, not treating it as an afterthought. Remember, data quality is a strategic asset, and following these frameworks transforms it from a technical issue to a business enabler.

As data manager in pharma, I often worked with two sets of data quality principles ALCOA and FAIR. ALCOA (Attributable Legible Contemporaneous Original Accurate) acronym was coined by FDA in 90s. In 2010s, four more principles (Complete Consistent Enduring Available) were added and it was renamed to ALCOA+, serving as a base framework for data integrity in Good Manufacturing Practices and Good Documentation Practices. FAIR framework, first defined in 2016 paper in Scientific Data journal, has already been widely adopted to make data Findable Accessible Interoperable and Reusable in the face of exponential increase of data volume and complexity. FAIR adoption can accelerate innovation, eliminate data silos and facilitate the use of AI.

Always consider the question: "Buy or Develop". Based on your budget, size of the organization, and business need, it may be easy for you to buy some SaaS (because there is a lot of variety of those on the market), which will be available immediately due to a plug-and-play solution. But some of the companies may develop their own solutions for DQ (including incorporation of popular open-source tools, like Great Expectations or deequ), because of legacy systems or desire to have more control over it.

Understand the diiference between DQ tools available currently in the market. There are two Sides of the DQ Coin. It gets messier with the platform landscape as no single vendor exists today that can do everything in terms of Data Quality. For example, Data observability tools tend to target the developer domain, hence they often find themselves portrayed as IT tools. But these tools might not quite hit the mark for our business enthusiasts. They might find these tools a tad less user-friendly and insufficient to serve up those critical insights, something that can be found in traditional DQ tools. Try to find the right balance between the two before selecting a tool to learn

Data quality management is greatly benefitted from the right tools and techniques. It's not just about efficiency; these tools provide accuracy, consistency, and reliability to our data quality initiatives. In-house tools like dashboards, rules engines, and scorecards are essential. They are often tailored to specific organizational requirements and integrate seamlessly into existing workflows. Using external tools, can also provide fresh innovative solutions. They can fill gaps in our toolset and introduce the best practices from a wider group. These tools are enabling us to not only execute but continuously enhance our data quality processes. The goal is to make data quality a seamless part of our business operations.

Since business and regulation evolve more rapidly than systems and processes, data quality is rarely considered from the project design phase and controls are normally detective, rather than preventative. From the non-embedding of data quality in the design phase, it can follow that fixing data quality can be difficult and expensive. A cultural change towards data quality-by-design is beneficial. Data quality programs typically start with a framework consisting of governance, ownership/stewardship, processes and organisation. Striking the right balance between under-governing data and pushing the organization to work around uncompromising rules, can be difficult. The right solution depends on the corporate-specific organization and culture.

I would suggest searching popular developer communities for Data Quality topics and, of course, the biggest Slack Community on that: Data Quality Camp

To ensure effective data management, it's crucial to keep your data quality skills up-to-date in this constantly evolving field. Staying informed about the latest tools, industry trends, and best practices is key to addressing data quality challenges. Here are some ways to stay current: 1. Enhance your foundational knowledge by taking online courses and tutorials and earning certifications from reputable organizations like Data Varsity, DCAM, or DAMA. 2. Join online forums and connect with experts to gain insights into the best practices used by organizations. 3. Create dashboards on platforms like Power BI to better understand your business's various data needs and make informed decisions about what to do with your data.

Sometimes joining network groups within an industry for the data management, data governance involved employees across functions can sometimes give a different perspective on your specific issue. Start your own or find an existing one. Ask around in the field of your industry if anyone is involved anywhere, and you can share knowledge with like minded people. Have a charter of confidentiality if you are industry competitors, or have a set of rules to avoid conflicts of interest. I participate in two network groups, one in my union for financial and banking employees and another with some of the same people but also people outside banking. They include e.g. data warehouse people, data stewards, data quality leads etc.

Learning from experts and peers is invaluable in the ever-evolving field of data quality. Engaging with data quality professionals within your organization or industry provides insights that are both practical and relevant. It's also crucial to tap into the wider community. Forums, groups, and events offer diverse perspectives and innovative approaches. They're a goldmine for best practices and real-world challenges. Remember, data quality is a collaborative field. Networking and sharing experiences not only enhance our own understanding but contribute to the collective knowledge base.

Some of the ways to improve your data quality skills are - Learn about different Data Quality Dimensions and make a dashboard to track these data quality dimensions as KPI's for different data products. - While testing for data quality write cross queries to test if questions written by data engineers are fine and working according to requirements. - Joins conditions within datasets should be tested thoroughly to identify whether they are working properly or not. - SQL and Python skills also come in very handy to identify and automate a lot of Data Quality test cases

Data quality is certainly a journey, not a destination. Applying skills to real-world scenarios is the most effective way to enhance them. Whether it's through audits, improvement plans, or even competitions, each experience adds a significant dimension to our understanding. Monitoring and monitoring progress is crucial. Metrics and reports are not only based on track performance, but also highlighting areas for improvement. Feedback, whether from peers or through formal assessments such as certifications, is essential for continuous growth. Remember, every project, every report, and every piece of feedback is a chance to learn and improve.

Nowadays, you can find many open-source tools to test multiple aspects of the data quality lifecycle. Here's a simple path you can follow to practice for free: 1. Generate fake data using Faker (Python library) or Mockaroo, or use open datasets from Kaggle. 2. Generate a profile report with ydata-profiling to understand the data and create data quality requirements. 3. Save the data in Postgres or DuckDB. 4. Use Great Expectations or Soda libraries to create data quality checks. 5. Run the data quality checks and analyze the results (Great Expectations provides an HTML report).

Learn by doing. Of course some basic knowledge is important but for example detecting impactful Data Quality issues is something you need to learn by trying it out: what questions are relevant to ask, how do I get business people to open up about the consequences of bad data (that they sometimes are the cause of themselves...), ..... Same goes with defining relevant DQ rules: first try to describe the rule in simple words and work from there, eventually with the help of a more technical person, to define a formula that could work to validate a certain data element. Do this iteratively (PDCA cycle) so you get better after every iteration.

In my view, the most effective way to refine data quality skills is through hands-on experience. While courses and seminars offer theoretical knowledge, diving into real datasets provides invaluable insights. Each dataset has unique challenges, and by actively engaging with them, you develop a keen sense for spotting inconsistencies. It's akin to a craftsman perfecting his art through repetition. Theoretical knowledge is foundational, but true mastery in data quality emerges when theory meets practice. So, immerse yourself, tackle real-world data issues, and learn by doing.

I would suggest reading a book: Data Quality Fundamentals by Barr Moses, Lior Gavish, Molly Vorwerck. It gives you a full and basic understanding of how DQ actually works and helps organizations.

An aspect to consider would be creating a knowledge base of the domain data entities. Linking this knowledge base to a master data management system would provide ease of master data management to data stewards.

One thing that I found essential to ensure the quality of the data is to not just stick with the standard ways or general checklist for testing your data, you need to design different test cases for different scenarios based on your understanding of the data. Also understand your business before jumping into executing your common test cases, this way you can reduce the UAT issues tremendously. Also it's always helpful to have acceptance sample or criteria to validate against during staging, transformation and loading phases

Listen to your data consumers, understand their needs. Let clear business, regulatory needs drive your data quality work. Don't create and spread generic rules all across, create value adding and functional rules, and consider the end to end. What will happen when those rules fail, will anyone care to solve the whatever issue was discovered? Good luck