Bridging the gap between data and strategy ✦ Head of Data Strategy @ Profusion ✦ Author of The Data Ecosystem newsletter ✦ R Programmer ✦ Policy Nerd
Not all data technologies are created equal, and not all should play the same role in your organisation. Its time we break these into categories by level of priority: 💻 𝐓𝐡𝐞 𝐅𝐨𝐮𝐧𝐝𝐚𝐭𝐢𝐨𝐧𝐚𝐥 𝐃𝐚𝐭𝐚 𝐒𝐭𝐚𝐜𝐤 – This base layer is the backbone of the data platform, providing the necessary infrastructure for data storage, processing, and visualisation - Data Storage - Data Processing & Transformation - Orchestration - ETL Tooling - Analytics, Consumption & Visualization Tools 🚀 𝐏𝐫𝐢𝐦𝐚𝐫𝐲 𝐒𝐜𝐚𝐥𝐚𝐛𝐥𝐞 𝐀𝐝𝐝-𝐎𝐧𝐬 – These tools are essential depending on your organisation’s needs, and enable scalability for growing data requirements (e.g., managing customer interactions, ensuring security, performing machine learning). - Cybersecurity & Privacy Tooling - Customer Data Platform (CDP) - Data Science Modelling - API Services/ Integration Tools 👀 𝐍𝐢𝐜𝐞 𝐭𝐨 𝐇𝐚𝐯𝐞𝐬 𝐭𝐨 𝐈𝐦𝐩𝐫𝐨𝐯𝐞 𝐐𝐮𝐚𝐥𝐢𝐭𝐲 – Data quality tooling has become increasingly popular. I list them as nice to haves because they should never be the initial investment. Before you buy and implement these technologies, you should have a working storage and processing layer accompanied by a Data Governance, Management or Quality team - Data Observability/ Quality Tooling - MDM (Master Data Management) Tooling - Data Catalogue & Lineage 🧠 𝐏𝐨𝐭𝐞𝐧𝐭𝐢𝐚𝐥 𝐅𝐮𝐭𝐮𝐫𝐞 𝐓𝐨𝐨𝐥𝐢𝐧𝐠 – These tools are not widespread. They can help facilitate and improve data literacy, driving usage and innovation if business stakeholders have enough time to learn how to use them. - Knowledge Graphs - Semantic Layers Understanding your tech hierarchy is essential to planning your tech stack and figuring out how they all work together. Unfortunately, most companies don’t do this So check out the article I wrote about building your tech strategy holistically and intelligently (newsletter link in the comments). This was also my 𝗯𝗲𝘀𝘁 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗶𝗻𝗴 𝗮𝗿𝘁𝗶𝗰𝗹𝗲 𝘀𝗼 𝗳𝗮𝗿, so definitely worth checking out! #dataecosystem #techstrategy #newsletter #datastrategy #dylandecodes
Very informative and insightful article, Dylan Anderson, one of real challenges facing tooling in data space is the complexity contributed from a number of interrelated factors, including legacy (or existing ecosystem), people, processes, integration/interoperability, changes resulted in by innovations, business needs, emerging concepts/approaches/architectures/technologies. The layered profiling or classification of technologies is a good presentation but might not be enough for enabling understanding, analysis, control and management of complex relationships and dependencies. Who is actually or should be responsible for tooling and change/evolution management among DE, DG, DM, and teams for strategy and digital transformation? Or, is this pile or complex stack of technologies just changing and evolving by 'itself' after data teams reactively address some integration and interoperability needs and process changes required? Are there any useful and updated playbooks for such a challenging and on-going task if DG/DM/EA frameworks do not address it?
It's a matter depending on your data maturity. You are right and for many organizations the first layer is often a recurrent challenge to stay here on a good level. But as you are writing, too, technology is only one side of the medaillon. Even is technolgy changes fast and even if AI will change everything, we still need people and processes. Congratulation for the successfull article Dylan Anderson !
Breaking data technologies into priority categories is a smart approach. The foundational stack is key for ensuring stability, but companies often overlook the need to build that base before adding scalable or niche tools.
I am looking forward to seeing how this is changing in 2025. I assume we will see more change in the foundational stack in the next 2-3 years. AI will do his work there.
Dylan This is very elaborate, and CDPs have always been a very hot topic, especially in the marketing domain. Where do you reckon governance should be placed here?
I love the fact that you used a pyramid to display that. The facts are that most companies only need a basic "Foundational Data Stack". You probably don't need more for many, many years into your journey.
Thank you for sharing this. Sometimes you need structures like this to clear the clutter.
Bridging the gap between data and strategy ✦ Head of Data Strategy @ Profusion ✦ Author of The Data Ecosystem newsletter ✦ R Programmer ✦ Policy Nerd
2d📌 This article goes back into some of the fundamental truths of data technology. Definitely worth a read! - https://thedataecosystem.substack.com/p/issue-19-developing-an-overarching