Foundational

Exciting release from Snowflake - column-level lineage! This certainly has a lot of people excited Read our thoughts below 👀

Check out our CTO Barak Fargoun’s latest post on why enums create persistent challenges for data teams—especially when data warehouses fail to grasp their semantics.

Exciting news! We're sharing today that Julien Le Dem is joining Foundational as our latest advisor. Being at the forefront of data lineage research is at the core of what we do, and we're excited to work with Julien and the OpenLineage community to introduce new ways of driving end-to-end accurate lineage and improving the collaboration between the different lineage platforms in the ecosystem. Learn more about the recent news here >> https://lnkd.in/eYtGc8xJ

𝗪𝗵𝗮𝘁 𝗪𝗲 𝗟𝗲𝗮𝗿𝗻𝗲𝗱 𝗳𝗿𝗼𝗺 𝗔𝗻𝗮𝗹𝘆𝘇𝗶𝗻𝗴 𝟱𝟬𝗞 𝗗𝗮𝘁𝗮 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲 𝗣𝗥𝘀? Foundational recently hit a major milestone: scanning 50,000 pull requests from data pipelines code. These PRs span a wide range of frameworks, including dbt, Airflow, Spark, Google Dataform, and ORMs like SQLAlchemy, TypeORM, Ruby Active Record, etc. 𝗛𝗲𝗿𝗲 𝗮𝗿𝗲 𝟱 𝗸𝗲𝘆 𝗶𝗻𝘀𝗶𝗴𝗵𝘁𝘀 𝘄𝗲’𝘃𝗲 𝗴𝗮𝗶𝗻𝗲𝗱: 𝟭. 𝗗𝗮𝘁𝗮 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲 𝗣𝗥𝘀 𝗮𝗿𝗲 𝗺𝗲𝗿𝗴𝗲𝗱 𝗳𝗮𝘀𝘁𝗲𝗿 𝘁𝗵𝗮𝗻 𝗽𝗿𝗼𝗱𝘂𝗰𝘁 𝗰𝗼𝗱𝗲 On average, PRs for data pipeline code are merged in under a day, compared to 1.5 days for regular product code. This trend is most visible in dbt and LookML repositories, which might reflects the ease of coding in these modular frameworks. 𝟮. 𝗨𝗽𝘀𝘁𝗿𝗲𝗮𝗺 𝘀𝗰𝗵𝗲𝗺𝗮 𝗰𝗵𝗮𝗻𝗴𝗲𝘀 𝗼𝗰𝗰𝘂𝗿 𝗺𝗼𝗿𝗲 𝗳𝗿𝗲𝗾𝘂𝗲𝗻𝘁𝗹𝘆 𝘁𝗵𝗮𝗻 𝗮𝗻𝘁𝗶𝗰𝗶𝗽𝗮𝘁𝗲𝗱 These upstream changes originate from product code modifications that alter the schema of OLTP systems like Postgres, which are later synced to data warehouses or lakehouses via ETL tools like Fivetran. We observed that around 5% of these upstream changes impact database schemas, directly influencing downstream pipelines. This percentage is higher than we initially expected. 𝟯. 𝗠𝗼𝘀𝘁 𝗰𝗵𝗮𝗻𝗴𝗲𝘀 𝘁𝗼 𝗱𝗮𝘁𝗮 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀 𝗵𝗮𝘃𝗲 𝗮 𝘀𝗺𝗮𝗹𝗹 𝗶𝗺𝗽𝗮𝗰𝘁 Small-impact changes are about twice as common as large-impact ones. 𝟰. 𝗕𝗿𝗲𝗮𝗸𝗶𝗻𝗴 𝗦𝗰𝗵𝗲𝗺𝗮 𝗖𝗵𝗮𝗻𝗴𝗲𝘀: 𝗔 𝗖𝗼𝗺𝗺𝗼𝗻 𝗜𝘀𝘀𝘂𝗲 𝗔𝗰𝗿𝗼𝘀𝘀 𝗧𝗲𝗮𝗺𝘀 Schema-breaking changes occur when columns are renamed or removed while still being used downstream, causing disruptions to downstream pipelines. Although these changes are rare within a single framework (like a single dbt project or Airflow repository), they are quite common across teams or different frameworks. For example, a change in one dbt project might break another dependent project, or an Airflow modification could affect dashboards in Looker or Tableau. 𝟱. 𝗦𝗤𝗟 𝘀𝘆𝗻𝘁𝗮𝘅 𝗲𝗿𝗿𝗼𝗿𝘀 𝗮𝗿𝗲 𝗺𝗼𝗿𝗲 𝗰𝗼𝗺𝗺𝗼𝗻 𝘁𝗵𝗮𝗻 𝘄𝗲 𝗶𝗻𝗶𝘁𝗶𝗮𝗹𝗹𝘆 𝗲𝘅𝗽𝗲𝗰𝘁𝗲𝗱 When we introduced SQL syntax error detection into Foundational’s analysis, we assumed these errors would be extremely rare. However, we quickly learned otherwise. Frameworks like dbt, Airflow, and Spark often wrap SQL without proper validation, making it not uncommon for PRs to contain syntax errors. These errors can be difficult to spot, especially when the SQL is generated using complex logic, such as intricate dbt macros.

Tristan Handy, CEO of dbt Labs, at the Coalesce Keynote: "Do upstream teams often break your data?" *everyone is nodding* He's right of course--as we've been saying all along! Meet us at booth 225 if you're in Vegas this week. #dbt #coalesce #dataquality #schemachanges

Tristan Handy, CEO of dbt Labs, at the Coalesce Keynote: "Do upstream teams often break your data?" *everyone is nodding* He's right of course--as we've been saying all along! Meet us at booth 225 if you're in Vegas this week. #dbt #coalesce #dataquality #schemachanges

Just about to get started here at #dbt #coalesce 2024 🥳 Wishing everyone a fantastic conference, participants and companies alike. Come meet us at booth #225! Foundational ❤️ dbt Labs

Classifying PII Data in Snowflake: Just Add ‘PROPAGATE = TRUE’? At the June Snowflake conference, I was thrilled to learn about an upcoming feature that could revolutionize how companies handle PII data. Snowflake is planning to introduce automatic label propagation for tags, directly addressing one of the most persistent challenges: identifying all instances of PII data. Once identified, enforcing the necessary data protection policies becomes much more straightforward. Although specific details were scarce, the concept is straightforward and promising. You create a tag, set it to automatically propagate (check out the attached screenshot), and Snowflake takes care of the rest. Every table, view, or object that interacts with this data will automatically inherit the tag. Combined with Snowflake’s existing classification capabilities, this could be a game changer for organizations grappling with PII data management. However, it’s important to note that this feature won’t extend to data outside of Snowflake’s environment. Companies will still face challenges with end-to-end governance of PII data—a space where other vendors, including Foundational, continue to innovate. Nonetheless, this represents a significant leap forward. Although Snowflake hasn’t set an official timeline, they did mention that a private preview will be available ‘soon.’ I’m looking forward to the official release and the impact it will have on the industry.

Excited to share that Barak and I will be coming to Big Data LDN! Looking forward to chatting about preventative data quality, data governance, and of course, what's next for data lineage! Please reach out to schedule some time ahead and learn more about Foundational, or schedule directly with us: https://lnkd.in/gj6qASYY We had such a great time last year and I'm really looking forward to another fantastic event!

A few months ago Snowflake released its own native data quality monitoring framework called ... Data Quality Monitoring. 😊 Using Snowflake's native data quality monitoring can be very powerful - it's faster, more efficient, and a lot more secure, but it may also seem daunting to set up and maintain against hundreds and thousands of tables. So, we wrote a guide! Read Barak Fargoun's Guide to Snowflake Data Quality to understand how to set up your first data metric functions (DMFs), allowing you to boost data quality for your Snowflake data. Find the guide here: https://lnkd.in/gi-qtkYk #snowflake #dataquality #dataqualitymonitoring #datacontracts