Eric Olmsted, Ph.D.’s Post

Strong data pipelines are crucial for successful value-based care. Prioritize raw records close to the source for reliability and maintain clear documentation directly in your data tables. This direct approach helps prevent discrepancies and ensures that data processing is transparent. Organizing your data effectively and setting up early fail-safes in the process can greatly enhance trust and efficiency in healthcare analytics. These practices are not just about managing data—they're about building a foundation for better patient outcomes and streamlined healthcare operations. #HealthcareData #Analytics #ValueBasedCare

Why is it worth investing in metadata quality and structure? Don't we already have problems delivering ROI as is? For anyone working with data, this seems like a reasonable objection. However, metadata really is the key to overcoming many of the headwinds data teams are facing. How? 👉 Adoption and Use of Data Products By viewing tables and reports as data products (slightly simplified), a data team can use aggregated and derived metadata to understand who is using what, and how it is being used. This can then be used to prioritize tasks (e.g., a report used by many takes a long time to load and is expensive. Based on the data sources, one can set up an aggregate to avoid joining many tables "on the fly" - which will reduce runtime and cost, as well as improve the end-user experience, a win-win). 👉 Proactive Approach to Changes/Migrations By understanding data flow and data usage, one can predict how changes will affect existing processes (e.g., dbt models that depend on another model that will be changed), and downstream systems such as PowerBI. Another important point is that one can also understand which users will be affected, making it easier for the data team to communicate proactively about "downtime" or changes that will require adaptation on the user side. Sometimes changes are necessary, but it is much less stressful to own the communication versus controlling a situation where something has gone wrong. 👉 Find Tables That Are Never Used By finding tables that are never used, one can delete them, saving storage costs and compliance/GDPR issues regarding the rightful use of data. 👉 Find Jobs That Create Tables That Are Never Used As a rule, unused tables that are not a product of ad-hoc analysis (this should be done in a schema that automatically deletes data after i.e. 7 days) will be produced with some frequency, and if the tables are not used, it follows logically that a job that produces the table is also useless and can therefore be stopped. 👉 Eliminate Unnecessary Processing There is a lot of talk about real-time, but in most cases it is sufficient to update data at a frequency so that yesterday's data is available. It is obvious that running a frequency of once an hour versus once a day (every 24 hours), i.e. 24 times as many jobs, will make a huge difference in cost. If you are not entirely sure how this looks in the company, you can, for example, look at the access pattern of a table to understand what time frame it is used and adjust the update rate. Typical easy optimizations here will also be to skip runs on weekends and holidays, when people are typically not at work anyway. By going from 5 to 7 days a week where jobs are run, you have a saving of about 28%. If a high update rate (1 time per hour or more) is actually necessary, then you can limit the update to within normal working hours. Let's say between 8:00-16:00, i.e. a saving of 16 hours per day, or 66%. Stay tuned for additional tips and tricks!

𝐃𝐚𝐭𝐚 𝐌𝐨𝐝𝐞𝐥𝐢𝐧𝐠: 𝐖𝐡𝐲 𝐎𝐫𝐠𝐚𝐧𝐢𝐬𝐚𝐭𝐢𝐨𝐧𝐬 𝐒𝐭𝐫𝐮𝐠𝐠𝐥𝐞 𝐭𝐨 𝐀𝐥𝐢𝐠𝐧 𝐭𝐡𝐞𝐢𝐫 𝐌𝐨𝐝𝐞𝐥𝐬 𝐰𝐢𝐭𝐡 𝐁𝐮𝐬𝐢𝐧𝐞𝐬𝐬 𝐄𝐱𝐩𝐞𝐜𝐭𝐚𝐭𝐢𝐨𝐧𝐬 Ever spent hours crafting the ‘perfect’ data model, only for the business to expect a fire-breathing dragon… but all you’ve got is a sleepy housecat? 🐱 Sadly, that's the reality in most cases! But why does this happen? Orgs are often challenged with certain hurdles that jeopardise their efforts to build & deploy data models effectively to meet business needs. In this post, Let’s talk about why so many of us have been there—ready to deploy a great model, only to find it didn’t align with what business actually expected. 𝐈𝐧𝐯𝐨𝐥𝐯𝐢𝐧𝐠 𝐦𝐮𝐥𝐭𝐢𝐩𝐥𝐞 𝐬𝐭𝐚𝐤𝐞𝐡𝐨𝐥𝐝𝐞𝐫𝐬 As Michael Brenndoerfer highlights, data modelling is rarely a solo endeavour. It typically involves various stakeholders—business analysts, data architects, database administrators, & end users—each bringing unique requirements & perspectives that must be aligned &integrated into the data model. But this diversity leads to the same term is often interpreted differently, creating a lack of semantic ambiguity, leading to confusion & miscommunication, and making the process of data modelling more challenging. 𝐀𝐦𝐛𝐢𝐠𝐮𝐢𝐭𝐲 𝐨𝐯𝐞𝐫 𝐝𝐚𝐭𝐚 𝐦𝐨𝐝𝐞𝐥 𝐨𝐰𝐧𝐞𝐫𝐬𝐡𝐢𝐩 Unclear ownership in data modelling can be a silent disruptor of model effectiveness. When no one takes the reins, responsibilities get fragmented, leading to inconsistent designs, duplicated efforts, and slower decision-making. Without a defined owner, business stakeholders often hold back on input, resulting in models that drift away from business goals & lose their analytic value. 𝐇𝐚𝐯𝐢𝐧𝐠 𝐭𝐡𝐞 𝐜𝐨𝐧𝐜𝐞𝐩𝐭𝐮𝐚𝐥 𝐦𝐨𝐝𝐞𝐥 𝐢𝐧 𝐩𝐥𝐚𝐜𝐞 Without a shared understanding, deploying a solid conceptual model becomes difficult, and organizations end up with data models that miss critical business concepts. As Frances O'Rafferty points out, skipping the conceptual model phase eliminates the shared, high-level view essential for aligning data with business expectations. Without this, teams lack a common language, leading to unclear, inconsistent models that can’t fully support business needs—resulting in misaligned data & lost insights. 𝐌𝐚𝐧𝐚𝐠𝐢𝐧𝐠 𝐝𝐚𝐭𝐚 𝐬𝐰𝐚𝐦𝐩𝐬 As orgs rapidly accumulate data in cloud storage, they risk ending up with a "data swamp" rather than a clean, organized "data lake." This complicates the integration with business processes & makes effective modelling a challenge. Without a clear grasp of existing data, teams keep collecting more for each use case, fueling redundancy and bloated datasets. This misalignment ultimately restricts the org’s ability to turn raw data into strategic insights that support business goals. What are the challenges that keep your org away from the expected data model outcomes? 𝐒𝐢𝐠𝐧 𝐮𝐩 𝐟𝐨𝐫 𝐨𝐮𝐫 𝐧𝐞𝐰𝐬𝐥𝐞𝐭𝐭𝐞𝐫: https://lnkd.in/dDT7FB3y

The missing piece to data democratization is more actionable than a catalog. Don't get me wrong: catalogs are very much needed to make data discoverable, which is a first step to making it usable. But catalogs are already around for a few years and I don't believe that we really have arrived at a mature state where data is available for everyone. So what are we missing? The full article can be found on my Medium: https://hubs.li/Q02FlMTh0 🎁 The rise of BI and self-serve BI has been focussing on maximizing the number of data consumers. Over time the dependency on data teams has shifted: first consumers were depending on them for insights, now for data. Yet those data teams do not always understand all business processes, and goals, and at larger organizations, they can not be expected to do so. That's why data products and a federated data workforce are emerging: to maximize the number of data producers, who have expert business knowledge on their domain. 🌐 Product thinking and federation are happening right now. Sometimes a bit pragmatic, for example first focussing on source-oriented data products, or less strict definitions of data products. Note that there is not yet a single definition probably. Yet what is clear already, is that data products are a concept that fit not directly in current catalogs. 🏛️💼📜 Some governance is missing and it is not limited to having an overview of what exist. Data sits across multiple systems, metadata is available in catalogs, compute still lives somewhere else. Navigating these tools and governing cross-tool processes, that's what missing and that's where the Data Product Portal is aiming to become the bridge. I'm keen to getting feedback! #dataproduct #datacatalog #datamesh

✨ 𝐏𝐨𝐰𝐞𝐫 𝐁𝐈 𝐈𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 𝐐𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬 𝐒𝐞𝐫𝐢𝐞𝐬 - 𝐃𝐚𝐲 𝟖✨ 𝟐𝟗. 𝐰𝐡𝐚𝐭 𝐢𝐬 𝐧𝐨𝐫𝐦𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧? Normalization refers to the process of organizing data within a database to reduce redundancy and improve data integrity. ➟ This involves dividing a large table into smaller, related tables and defining relationships between them. ➟ The goal is to ensure that each piece of data is stored only once, making the database more efficient and easier to maintain. ✅ Key Points about Normalization: Reduces Data Redundancy Improves Data Integrity Enhances Query Performance 𝟑𝟎. 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐝𝐞𝐧𝐨𝐫𝐦𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧 ? Denormalization involves combining multiple normalized tables into fewer tables, usually to improve query performance and simplify data retrieval. ➟ While normalization aims to reduce redundancy and improve data integrity by dividing data into related tables, denormalization intentionally reintroduces some redundancy to optimize read-heavy operations. ✅ Key Points about Denormalization: Improves Query Performance Simplifies Data Retrieval Increases Redundancy 𝟑𝟏. 𝐰𝐡𝐢𝐜𝐡 𝐬𝐜𝐡𝐞𝐦𝐚 𝐢𝐬 𝐦𝐨𝐫𝐞 𝐧𝐨𝐫𝐦𝐚𝐥𝐢𝐳𝐞𝐝 𝐬𝐭𝐚𝐫 𝐬𝐜𝐡𝐞𝐦𝐚 𝐨𝐫 𝐬𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞 𝐬𝐜𝐡𝐞𝐦𝐚? The snowflake schema is more normalized than the star schema. Example to Illustrate the Difference: ✅ 𝑺𝒕𝒂𝒓 𝑺𝒄𝒉𝒆𝒎𝒂: 📍Fact Table: Sales (SaleID, DateID, CustomerID, ProductID, Amount) 📍Dimension Tables: Date (DateID, Date, Month, Year) Customer (CustomerID, Name, Address, City, State) Product (ProductID, ProductName, Category, Price) ✅ 𝑺𝒏𝒐𝒘𝒇𝒍𝒂𝒌𝒆 𝑺𝒄𝒉𝒆𝒎𝒂: 📍Fact Table: Sales (SaleID, DateID, CustomerID, ProductID, Amount) 📍Dimension Tables: Date (DateID, Date, MonthID, YearID) Month (MonthID, MonthName) Year (YearID, Year) Customer (CustomerID, Name, AddressID) Address (AddressID, CityID, StateID) City (CityID, CityName) State (StateID, StateName) Product (ProductID, ProductName, CategoryID, Price) Category (CategoryID, CategoryName) ➟ Star Schema: Less normalized, simpler structure, potentially faster for query performance due to fewer joins. ➟ Snowflake Schema: More normalized, more complex structure, better for reducing redundancy and improving data integrity. 𝟑𝟐. 𝐰𝐡𝐢𝐜𝐡 𝐭𝐚𝐛𝐥𝐞 𝐜𝐨𝐧𝐭𝐚𝐢𝐧𝐬 𝐦𝐨𝐫𝐞 𝐚𝐦𝐨𝐮𝐧𝐭 𝐨𝐟 𝐝𝐚𝐭𝐚 𝐚𝐦𝐨𝐧𝐠 𝐝𝐢𝐦𝐞𝐧𝐬𝐢𝐨𝐧 𝐚𝐧𝐝 𝐟𝐚𝐜𝐭 𝐭𝐚𝐛𝐥𝐞? Ans: Fact Table ➟ This is because fact tables store transactional data, with each row representing a specific event. ➟ These events accumulate rapidly, leading to a large number of rows. ➟ In contrast, dimension tables store descriptive attributes about entities (e.g., customers, products), resulting in fewer rows. Shashank Singh 🇮🇳 Munna Das Naveen Kumar Adapala Srinath Ankolla Thodupunuri Bharath MADHU THANGELLA Santosh Pothnak 💎 Nomaan Ali G Bharath Kumar Keerthana S #powerbi #interviewquestions #dataanalytics #datamodeling #businessintelligence

The concept of "data product" has evolved, along with the principles of "data as a product" introduced by Zhamak Dehghani in the context of data mesh. This principle is sometimes shortened to "data products," which can lead to confusion. Xavier Gumara Rigol in his article Data as a product vs data products. What are the differences? clarifies the differences between Data as a product and data products . A must read one! To understand the definition of "data as a product" in the data mesh world, this quote from Zhamak Dehghani’s original article is key: “Domain data teams must apply product thinking […] to the datasets that they provide; considering their data assets as their products and the rest of the organization’s data scientists, ML and data engineers as their customers.” In summary, "data as a product" is the outcome of applying product thinking to datasets, ensuring they possess capabilities such as discoverability, security, explorability, understandability, and trustworthiness. Data product” is a generic concept and “data as a product” is a subset of all possible data products. A Data product is a product whose primary objective is to use data to facilitate an end goal.” Data products, whether they are entire customer-facing products or partial back-end products, possess different characteristics than other technology products. Simon O'Regan devides data products into the following categories • 𝗥𝗮𝘄 𝗱𝗮𝘁𝗮: Data collected and made available as it is, with minimal processing. • 𝗗𝗲𝗿𝗶𝘃𝗲𝗱 𝗱𝗮𝘁𝗮: Data that has been processed to some extent, with additional attributes or transformations applied. • 𝗔𝗹𝗴𝗼𝗿𝗶𝘁𝗵𝗺𝘀 : Services where data is input into an algorithm, which then processes the data and returns insights or information. • 𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻 𝘀𝘂𝗽𝗽𝗼𝗿𝘁: Information and insights are provided to assist users in making decisions, with most processing done on the provider’s side. • 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗲𝗱 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻-𝗺𝗮𝗸𝗶𝗻𝗴: Systems where all of the intelligence and decision-making within a given domain are automated. The algorithm performs the work and presents the user with the final output. Must read articles Designing Data Products https://lnkd.in/d_xawwB8 How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh https://lnkd.in/dWAYrhqB Data as a product vs data products. What are the differences? https://lnkd.in/dDjdJmES #dataproduct #dataasaproduct #productmanagement

The concepts of Data Products, Data as a Product, and Data Assets are foundational in data management and analytics. Each structure highlights a different perspective on how data can be utilized and valued within an organization or the marketplace.  A lot of confusion exists about the definitions of each structure.     I have my own perspective on what qualifies the use or collection of data as a Data Product, Data as a Product, or a Data Asset. The long version is available on Substack. Here is a quick preview: Data Product: A collection of data curated and managed with a product lifecycle. It leverages predefined data and data patterns to solve a problem or fulfill a specific need for the consumer. Data as a Product: Refers to the ecosystem of data as a holistic product that can be accessed with a variety of methods. It is the buffet of data. DaaP is differentiated by the meticulous application of mature data management practices and well mapped and understood data connections, definitions, and sources of truth. It shifts the focus from the mere collection of data to curating and presenting it in a manner that adds direct value. Data Assets: The concept of data assets emphasizes the strategic value of data, treating it as a key resource that can provide a competitive advantage when managed and utilized effectively. Data assets are considered an integral part of an organization's intellectual property and are protected and managed with care to ensure they maintain their value. Check out the full article on Substack. Read, Like, Follow

Developing a Scalable Data Ecosystem If you’ve ever participated in a meeting where the accuracy of data points was in question, you are certainly not alone. Common obstacles such as inconsistent metrics, disparate definitions, and ambiguous data flows often hinder teams striving for data-driven decision-making. What is the solution? The answer lies in creating a cohesive ecosystem where tools such as data catalogs, semantic layers, business glossaries, ontologies, and observability can integrate seamlessly. Initiate with the Data Catalog The data catalog serves as the foundation of a robust data system. It is essential for organizing all metadata, tracking data lineage, and facilitating the discovery of data assets. A well-structured catalog additionally supports: Business Glossaries: Precise, standardized definitions Ontology: Defined relationships between concepts Lineage: The capability to trace a metric back to its origin Incorporate a Semantic Layer for Cohesion The semantic layer enhances the accessibility and reliability of data for business teams. It translates raw data into business-relevant metrics (such as "Gross Margin" or "Monthly Active Users") and ensures consistency across platforms like Power BI, Tableau, or Looker. With a semantic layer, you will find that: Analysts can cease writing SQL for each new inquiry. Metrics are defined once and maintained consistently across the board. Governance is inherent—preserving the security of sensitive data. Prioritize Observability No system is immune to failure without proper monitoring. This is where observability comes into play, acting as a health check for your data: Pipeline Monitoring: Guarantees that your ETL/ELT processes operate seamlessly. Data Quality Checks: Ensures your data aligns with the business rules. Metric Alerts: Identifies anomalies when a KPI appears inconsistent. Steps to Initiate Invest in a Data Catalog: Start the organization of your data assets with platforms such as Alation, Collibra, or Purview. Define Metrics in a Semantic Layer: Harness solutions like dbt Metrics or AtScale to centralize and standardize key KPIs. Implement Observability: Incorporate tools that monitor data quality and pipeline performance. Collaborate on a Glossary: Partner with both business and technical teams to articulate essential terms and relationships. The Importance of This Approach Organizations that effectively connect these elements experience fewer disputes regarding data, accelerated insights, and enhanced trust in the information. This benefits all stakeholders, analysts spend less time seeking data, and decision makers receive accurate answers more promptly. How does your organization address these challenges? Are you working on establishing a comparable ecosystem? #DataAnalytics #SemanticLayer #DataGovernance #DataCatalog #Observability #BusinessGlossary #dataproduct

We need to rethink how we manage, use and leverage #Data! This has been a bug bear of mine for 20 years. We have grown up in 3Vs of (Big) Data, #Velocity, #Variety and #Volume, to which some added one or two more #Veracity and #Value. The problem with those 3 Vs was they were all about 'managing the data', and that is the issue I have with the two primary Data industry frameworks, #DAMA and #DCAM. These frameworks all start with how you store, manage and curate data, and give little or no thought to how it's leveraged. 🚦 It's not about Managing Data, it's about how we drive value from Data, and managing Data to drive that Value! 🚦 ☯ Both DAMA and DCAM now recognise #Analytics, but it's like something from a parallel Universe, not the Yang, to the Yin of Data. Likewise CIOs, CTOs, Architects and Data professionals still treat Data as primarily the 'exhaust' of Process and Technology. 🔦 Let's get back to Computing 1 - 0 - 1 ergo Input - Process - Output. When we realise that Data is the Input and the Output we can start to think of a Data driven Operating Model and Architecture, not just managing 'exhaust' Data! To start a new approach I propose to replace the Data Vs, with Data 4 Us. 😀 1️⃣ #Useful - getting Value from Data is like panning for Gold - there is an awful lot of 'silt' and a few nuggets. This is true for most organisations, whose Tera/Peta/ExaByte Lakes/Warehouses contain at least 90% of data that is no use whatsoever! Stop collecting, managing and storing Data that is not remotely useful! Collect the minimum you need and can use and leverage! Is that Data an Input and an Output. 2️⃣ #Unique - is the data you hold different and unique to what others are holding. Most Customer Data will be similar to what other organisations can collect, but if you can combine data from your systems, with third party sources, it is possible to create Unique data and this can be really valuable 🥇 in creating new insights, products and services for your customers! And in compute terms, if you have Unique Inputs, you can create Unique Outputs! 3️⃣ #Universal - is your data available across the entire business, with clear definitions. Too often you are dependent on the Data Models of the many, many Enterprise Software Vendors, meaning you end up with a complex Data landscape, #Mesh or #Fabric, with multiple definitions for Revenue, Customer, Product, as a result of systems complexity! Make your #data, normalised, standard and universal and drive the systems from the Data, not the Data from the Systems! 4️⃣ #Unified -does your data link to other data, does it connect to Analytics to drive towards a uniform, continually improving enterprise operating model and architecture. Am I making sense, or just barking up the wrong tree? Bill Schmarzo Vin Vashishta Mark Stouse Randy Bean Pascal Clement Jon Cooke Sol Rashidi Liz H. Daniel Rolles David Pidsley Douglas Laney

Granularity, referred to as the level of detail of our data, plays a key role in shaping the structure and functionality of our data models. Any data analyst or data engineer can relate to working with granularity shifts, that is, needing to accommodate the level of detail of the data from source to destination by, for instance, aggregating it. A higher granularity means more detail; whereas a lower granularity means less detail. Data Vault 2.0 won’t be the exception, as granularity takes on a unique significance, influencing how we capture, store, and deliver insights from our data. In traditional Kimball’s dimensional modeling, granularity is defined by the level of detail within fact tables. Here, we encounter three distinct granularities, from the highest level of granularity to the lowest they can achieve: transaction fact tables, period snapshot fact tables, and accumulating snapshot fact tables. Transaction fact tables encapsulate singular measurement events, typically having the highest level of granularity as they capture individual events or transactions in detail. Period snapshot fact tables summarize data over standard periods, such as days or weeks, offering a broader perspective suited for trend analysis. Lastly, accumulating snapshot fact tables track only deltas after an initial snapshot, ideal for monitoring changes over time. Now, how does Data Vault 2.0 approach granularity? In Data Vault, the Raw Data Vault serves as the custodian of raw data, preserving the original granularity from source systems. Unlike traditional models where aggregation might occur during loading, the Raw Data Vault maintains the finest possible grain, ensuring no loss of detail. While the Raw Vault efficiently organizes and integrates our data into hubs, links, and satellites, it maintains the granularity intact, serving as a bedrock for downstream transformations. However, in the journey from raw data to actionable insights, the Information Delivery layer often demands a different granularity tailored to the needs of the business, that is, our target granularity. Thus, arises the need to derive or transform the target granularity from the existing Raw Data Vault model, which will happen in the Business Vault and Information Mart layers. The transformation process will involve understanding business requirements, extracting meaningful insights, and aligning them with the structural integrity of the Data Vault. The goal is to bridge the gap between raw data and information delivery, aimed to provide tangible business value. In essence, DV2.0 maintains the finest possible grain within its Raw Data Vault, preserving the original level of detail from the source systems. This is done to keep the untouched history of the data, and hence being able to track anything back to its source (auditability). Then, the target granularity will be achieved in the downstream layers, such as the Business Vault and Information Mart.