Exciting release from Snowflake - column-level lineage! This certainly has a lot of people excited Read our thoughts below 👀
Column-level Lineage is now officially supported by Snowflake! (Official link in the comments) A few days ago Snowflake announced that column-level lineage is now available in Snowsight as a Preview Feature. This was long awaited and certainly got a few folks (including our team here at Foundational) very excited. IMHO it’s interesting for a few reasons: * There’s now an “official” reference for data lineage information you see in other tools or try to build yourself with open source parsers. While most tools supporting data lineage parse the same information (query logs), there are certainly big differences in accuracy, so it’ll be interesting to see if data catalogs will now pull this info directly, instead of parsing query logs for themselves, like they do for Databricks => My take: We’ll see most data catalogs switch to pulling Snowflake lineage info since it doesn’t make sense to parse on your own and risk having those differences. In that sense, this release also helps vendors and not just Snowflake customers. * Many Snowflake customers today *without* column-level data lineage can now get it from a credible source, and don’t need to buy a tool for lineage purposes only. But please note that the big deal about lineage is understanding cross-platform lineage (e.g., impact on dashboards), which is not supported here. Furthermore, this is for Snowflake Enterprise customers only, so I’d also assume that those larger companies already have a solution, but it’s still a meaningful alternative now. => My take: This puts some pressure on tools like dbt and data catalogs to become better in BI coverage and provide that “end-to-end lineage” experience everyone needs. Ultimately lineage is still a facilitator for a specific use case, for example cataloging, or CI/CD, which is our expertise at Foundational. It becomes more about what you do with lineage vs. do you have lineage. What are your thoughts? #datalineage #governance #snowflake