Unified Data Engineering: Consolidating Engineering Functions to Harness Artificial Intelligence

Mark your calendar. 2024 is the year that data integration will die. It has been 30 years since the number one vendor in data integration was founded.

Now is the Time to Implement Unified Data Engineering

The next ten will be the demise of data integration, with modern technology replacing a generation of legacy systems. There are three reasons why unified data engineering will emerge as the replacement of what has become a very splintered market.

REASON ONE

It is necessary. The need for a data integration replacement stems from 2022 EMA research showing that the average enterprise maintains at least 6-8 different data integration or data movement technologies. The splintered market includes individual platforms for data integration, replication, preparation, API integration, streaming data, and messaging, along with separate platforms for data lakes or cloud technologies. Every additional platform adds additional cost, complexity, and constraint; and most modern platforms lack built in governance and rich metadata services. Unified data engineering has the potential to replace multiple traditional platforms with a single solution.

REASON TWO

It is unprecedented. Data management has remained relatively unchanged for almost 25 years. This is the first time in the last 30 years where technology innovation threatens to replace traditional data management platforms. We have seen four waves of change, culminating in this unique opportunity. The big data wave showed us the importance of semi-structured data, making data integration tools obsolete. The cloud wave showed us the importance of distributed computing and universal access to data, making way for new tools. The generative AI wave with rapid ubiquity makes the need for solid data foundations absolutely necessary. Finally, unified data engineering redefines the way that organizations deliver data products in the modern world. Now is the time to make the shift to unified data engineering.

REASON THREE

It is urgent. Organizations that used to operate independently are now part of complex business ecosystems comprising significant relationships with suppliers, customers, partners, and investors. Orchestrating these complexities is vital to the success of the business. Since the speed of business will continue to move faster and become more complex, it is urgent to align business orchestration with unified data engineering to address today’s needs and to prepare for a faster future. Now is the time to make the shift to unified data engineering.

The Requirements for Unified Data Engineering

Unified - Disrupting Decades of Single Purpose Tools

The last three decades brought several major shifts in data and analytics technologies; from data warehouses and data lakes, to managed services and SaaS, to data centers and cloud. Vendors developed new types of data integration tools to address every new shift. The result is a plethora of diverse data management tools. To address the issues created by legacy and niche data management approaches, unified data engineering must bring the full set of data engineering functions together in a single platform. It must support all data types, at all latencies, for all use cases, in all locations.

All data types. Unified data engineering addresses the needs of all different types of data, with the ability to combine diverse data types including both structured and semi-structured data. For example, e-commerce platforms produce JSON files mixed with rich text and digital images. It must be easy to combine these data formats with structured data, without having to use multiple tools. The combination of multi-structured data provides richer insight and more complete context for the use of artificial intelligence and business insight.

All latencies. Unified data engineering processes both streaming and batch data, with the ability to combine data from both latencies in a single data pipeline. For example, a call center technician needs access to both historical data and realtime transactions to help a customer who calls in for guidance just seconds or minutes after making a new purchase. The connection of multi-latency data lays the foundation for immediate intelligent responses to business events as they occur.

All use cases. Unified data engineering covers a broad range of data use cases, including data collection, movement, replication, CDC, integration, quality, master data, and transformation, making it the perfect platform for the unification and consolidation of legacy data management platforms. For example, data coming from a network of devices on the internet of things requires cleansing of noise from the data, extraction of critical data, transformation to a consistent format, and integration with historical data for context. The amalgamation of multiple use cases in a single platform makes data pipeline automation and optimization more accessible for small and medium enterprises.

All locations. Unified data engineering provides access to data everywhere including SaaS, IoT, cloud, multi-cloud, on-premises, and hybrid data configurations, making it the platform for modern data ecosystems where data moves in and out of systems in all directions. For example, smart cars capture data from on-car sensors. Captured data can be stored, prepared, and analyzed in the cloud, with insight feeding into SaaS engineering platforms for product design or automating actions within the vehicle itself. Singular data engineering from collection to automation operationalizes machine learning, AI, and other actionable information with minimal effort.

Orchestrated - Disrupting Deficient Data Management Tools

The evolution of data over the last 30 years has spawned numerous modern technologies and even more opportunities for the exploitation of data. However, as new opportunities emerged, purpose-built tools were designed to address new data types, data storage, and data locations. The result has been tools designed, marketed, and oversold to do specific tasks in unique environments. Each individual tool has expanded capabilities but lacks the foundational architecture to support legacy or future shifts in data technology. To address the issues created by single use case tools, organizations should pursue a unified data engineering platform. To achieve unification of data engineering the new platform must be centralized, visual, intelligent, automated, metadata-driven, governed, optimized, and reusable, providing quicker access to more accurate data.

Centralized. To be truly centralized, a core command component must support a full set of administrative tasks to unify data engineering. Centralized features should include design, testing, operations, observation, automation, and optimization for all data pipelines.

Visual. While many data engineers prefer to code for data pipelines, modernization should include a drag and drop design interface for data engineering. Support for a low code or no code approach to development of complex data pipelines provides access to users without experience in data engineering.

Intelligent. With the explosion of generative AI in 2023, unified data engineering must include a rich set of AI-enablement using historical data to make recommendations on next best actions, potential opportunities, and potential risks. Gradually, AI will advance to include co-pilot capabilities, lowering the need for human intervention in the optimization of data flow throughout organizations.

Automated. Unified data engineering automates formerly manual and menial tasks in data engineering enabling data scientists and engineers to focus more time on value creation and innovation. Automation will continue to expand beyond basic actions to include operational improvements, risk aversion, and self-healing.

Metadata-Driven. A rich set of metadata is the key to the unification, intelligence, and automation of complex data engineering environments. Unified data engineering vendors with the most strategic collection and use of metadata will be the winners in terms of extensive unification and the use of artificial intelligence in their platforms.

Governed. Unified data engineering must be fully governed. When rich metadata is automatically generated, it streamlines the process of delivering on the promise of enterprise data governance and improves compliance measures. With the continually increasing volume and complexity of data, along with the growing prevalence of data sharing inside and outside the organization, the automation of governance is extremely important.

Reusable. A strong unified data engineering platform will separate data pipeline logic from execution and build intelligence into a data hub for maximum reuse of code. The best platforms will maintain reusability percentages as high as 80%, making future migrations nearly effortless and ensuring maximum leverage of current investments.

Optimized. The most mature unified data engineering platforms use sophisticated optimization algorithms to address issues like break-fix and resource conflicts when multiple, even thousands of data pipelines run concurrently in complex businesses. As previously mentioned, these platforms will quickly move to self-optimization and self-healing.

Platform - Designing the Future of Data

For the last few decades software features and functions have taken the spotlight with most organizations making buying decisions based on a set of capabilities they deem necessary and advantageous. Product architecture has taken a back seat, with most software architectures mirroring current trends around compute, storage, and networking technology. The result has been a ten-to-fifteen-year lifecycle for each new architecture, after which the former architecture renders the software obsolete. To address the issues created by short-sighted architectural decisions, unified data engineering takes a platform approach, building on a foundation designed for long-term viability regardless of shifts in infrastructure technology. Therefore, the unified data engineering architecture must be enterprise-ready, cloud-first, secure, available, recoverable, and automated.

Cloud-First. The domination of the cloud demands that vendors develop their unified data engineering platforms entirely for the cloud, with consideration that it might also need to work well on-premises. For hybrid use cases, there must be an integration of administration. To qualify as cloud-first, the platform should be serverless, elastic, boundless, and fully integrated.

Enterprise-ready. The modern enterprise operates as a business ecosystem that requires both internal and external stakeholders to operate as one. Along with global accessibility, an enterprise-ready platform must be secure, available, recoverable, and automated.

Secure. Unified data engineering makes security a requirement; security is built-in, not just tacked on. The most mature platforms will ensure that security must be programmed for all data engineering tasks, with rules and requirements built into the development cycle. Encryption is necessary for both data in motion and data at rest, something that is not easily achieved.

Available. As data pipelines become the lifeblood of the organization, flowing all throughout the business, data engineering must always be up and running. Automated security is even more vital as AI and data automation continue to take more extensive roles in data engineering.

Recoverable. Data loss is not an option. Every transaction carries potential value and modern transactions expire over time. Since unified data engineering handles all kinds of data at all latencies, recoverability must be extensive and programmable to fit the requirements of different types of transactions.

Automated. Automation is as important for the underlying platform functionality as it is for data engineering. Organizations do not have the time or resources necessary to run a business and be infrastructure experts as well. Ideally, platform automation allows complex organizations to operate as one, little to no consideration of underlying infrastructure maintenance and administration.

The Seven Benefits of Unified Data Engineering

While some organizations may hesitate in making the shift to unified data engineering, early adopters will quickly realize gains in both value creation and efficiency.

Data Engineering Transformation - from cost center to value creation

1.      Faster time to AI value. With data engineering taking up 75% of every AI or analytical project, most companies need to take out a construction loan to operationalize insight. They pay for the insight several times before they produce insight that yields a return on their investment. With data transformation, movement, and integration on a single platform, companies can expect to reduce time to value for their analytics and AI by up to 50%. Consider what it would be like to produce a single data pipeline that captures all the necessary data, automates the analysis, and delivers insight to both decision makers and machines without manual intervention.

2.      Increased AI value creation. Iteration is the key to improving AI and analytical accuracy, especially when it comes to predictive and prescriptive models. However, when data engineers, data scientists, and data analysts all use different tools to prepare and analyze data, the process slows down. By simplifying the delivery of analytics, companies will be able to iterate faster, continually improve analytical outcomes, and create more value using analytics and artificial intelligence.

3.      Competitive AI and analytics. Most organizations spend high dollars preparing data and utilizing machine learning to try and differentiate their business. However, due to technology and resource constraints, only a small percentage of organizations differentiate themselves based on analytical advancements. Unified data engineering enables companies to combine analytics, predictive models, and generative AI in new ways to create more competitive analytics not possible for their competition.

4.      Accelerated innovation. Innovation is the new oil; and the speed at which a company innovates separates extraordinary companies from the ordinary. Increased speed of innovation cycles gives organizations the ability to dominate and disrupt markets based on accurate intelligence. So, imagine a company orchestrating data on a single platform. Two things will happen. One, innovation takes place in cycles; those cycles will complete in record time because they no longer move data from platform to platform to deliver insight to the front lines. Two, innovation will take place in two different arenas: business and analytical innovation. Organizations that unify data engineering will create new business models and deploy new analytical models at a faster pace than the competition.

Data Engineering Efficiency - from 75% of analytical projects to 25%

1.      More strategic resource allocation. Every data engineering organization is being asked to do more with less. Successful teams find ways to optimize the use of their time for the greatest analytical return. Unified data engineering consolidates data management platforms, freeing up to two-thirds of your data engineering team to work on more strategic projects and shifting the focus of data engineering from data preparation to data science. In addition, more meaningful work for data engineers increases their commitment to your organization, reducing churn and increasing productivity.

Resource allocation also flows out to the rest of the organization. From an IT perspective, there will be cost savings from more efficient use of computing and storage resources, especially with Cloud Unified data engineering. Business analysts, business users, and executives will also save time finding and processing insight to make decisions. The result is better decisions faster.

2.      More optimal reuse. There will always be a new data migration. With the speed of innovation constantly increasing, we can expect the next migration to come sooner than the last. It is Moore’s Law applied to data management technology. The unification of data engineering on a flexible software architecture allows organizations to deploy once and use the code many times, guaranteeing up to 80% reuse of all code in future migrations.

3.      More seamless business alignment. The latest trend in strategic business management theory focuses on business orchestration, with leading companies creating strategic positions for orchestrators skilled at making several moving parts of the ecosystem work more efficiently together. With all data and analytics unified in a single platform, data engineers can orchestrate their projects in unison with business orchestration. Technical teams more easily align their work with business requirements and objectives, making themselves invaluable to the business.

The Search for Unified Data Engineering

Some software vendors are already aligning themselves with the requirements of unified data engineering. The superscalars, especially Amazon Web Services (AWS) , Google Cloud , and Microsoft Azure, are building unified data engineering into their cloud platforms. Data management companies like Informatica and Nexla , are completely remaking themselves with new cloud-first versions of their technology. Data streaming platform vendors like Confluent are enhancing their streaming processing and integrating a complete data governance solution to create a central nervous system for data that connects all operational and analytic data processing systems across the enterprise. Other streaming platform vendors, like StreamSets Inc. and Striim, see the data pipeline as core to the enterprise and expanding their transformation and AI capabilities. API, iPaaS and APIM vendors, like Boomi, MuleSoft, SnapLogic and Workato, are upping their investment in active use of metadata and expanding their capabilities to include more data engineering use cases. Cloud-migration data integration companies like Fivetran and Matillion are working to retrofit their platforms to address both modern and legacy requirements. Newer companies like Datavolo come from a background of streaming and integrating massive and multimedia files to embrace data management engineering, as well.

Stay tuned for a more detailed analysis of which vendors are making the right investments to win the race.

For more information, contact John Santaferraro at [email protected]

* First published at ferraroconsulting.com on January 22, 2024.