The true cost of the 'data rush'

by Robert Heeg

This article appeared, in full, in ESOMAR’s Global Market Research 2022 report.

The awareness has been growing that there has to be some kind of data management so stakeholders have the right data available in the right place, at the right time. A study by Accenture and HfS Research showed in 2018 that nearly 80 per cent of financial services organisations are experiencing an influx of unstructured data. More importantly, perhaps, is that most of the participants in this study indicated that 50 to 90 per cent of their current data is unstructured.

Technology-financial rush

Artificial intelligence (AI) has played a key role in analysing unstructured data, and a firm data and technology strategy is now part of a typical enterprise AI roadmap. But this required investment first, tells Joaquim Bretcha, Ex-Officio ESOMAR President. “Our industry has been immersed in the combination of technology-investment-startups that has characterised the global economy of the last decade. In this technology-financial rush, many start-ups have developed capabilities that have proven useful for the market research industry. The need to understand people's behaviour and opinions is endless. While people's life is getting more intense in the digital arena, all technologies will keep developing.”

For many organisations, a cost-effective approach to unstructured data management is to choose a third-party partner or vendor. There is an increasing number of companies specialised in solving data management and data analytics issues at scale. A lot of innovation is happening here in data search, data analytics, and data intelligence. This vendor-based technology allows enterprises to benefit from tech’s best practices, experience and implementation expertise, especially in the larger language models.

As most companies dedicated to analysing unstructured data are start-ups or early-stage companies, the question is: which place will they occupy in the international competitive arena? Michalis A. Michael, CEO of DMR, thinks they will play a key role. “If we consider that most data available for analytics and insights are unstructured, then it is safe to predict that they will either take over from the big analytics companies or will be acquired by them.” Bretcha predicts a cycle of concentration and integration. “Those start-ups or early-stage companies capable of developing smart solutions will be attractive to larger established groups. These can either be market research agencies, consultancies, or big platforms in need of better understanding their customers.”

Data gold rush

As analysing unstructured data is largely the domain of AI, one may wonder what the role is of the human in this world of automation and higher complexity in analysis. Hana Huntová, executive director at SIMAR, is clear about this: “A machine is a machine. It only does what the humans ask it to do.”

She likens the current state of affairs to a ‘Klondike Gold Rush for data’. “The qualities of teams who actually find the golden nuggets are emerging. And the issues with mining are annoyingly similar to the problems of the past. Do we have the right tools? Are we asking the right questions? Do we understand enough of what the client business needs to know? In a more complicated data environment, it is even more important that there is somebody with a helicopter view: someone who knows enough, but perhaps not everything, about every approach and concept but who still is able to drive the team toward a conclusive result.”

She underlines the need for somebody who keeps the strategy in place and deploys skillful experts to perform their advanced explorations. The varied skills in a team and their effective cooperation are more important than ever, says Huntova. “In the past, case studies were built on one winning methodology, superseding everything else. But perhaps the future is more of a patchwork.”

Bretcha agrees that the role of the human is and will be key. “Machines need to get connected with machines in order to maximise their capacities. And humans must control and lead their performance. Machines enhance and enrich people's capacities but cannot replace them. We people are so complex that the logic of machines cannot be left alone for a long time without being trapped in our contradictions. The human nuances and deep motivations must be managed by people with the help of the power of machines.”

Contradictions and paradoxes

Despite its great promise, unstructured data also has its limitations. The cost has been a major issue; according to Krishna Subramanian, president and COO of Komprise, more than two-thirds of the cost of data in Venturebeat is not in the storage but in its active management. He knows that for every piece of data, organisations typically keep several backup copies as well as a replication copy for ‘disaster recovery’. Subramanian calculates that if your data is growing at 30 per cent, it’s more like 90-100 per cent when all the copies of the data are factored in.

Another challenge lies in our human contradictions and paradoxes, observes Bretcha. “This can be overcome by combining the power of machines with the intelligence and creativity of people.” Michael identifies another limitation: “The accuracy of annotation in the many languages. Especially when it comes to audio, there is an option to annotate content directly or transcribe to text, and that adds another layer of inaccuracy creep.”

The problem of the unstructured data is not in the data itself, thinks Huntova. In her opinion, the successful deployment of such data is in the precise definition of the research aim and then finding a team with balanced and complex capabilities that has a somewhat eclectic mix of methods and tools at hand.

“I once worked within a marketing team. We had so much data available to us and so much different expertise and experiences; we had data from consumer marketing, trade marketing, sales, logistics, finance and research. Once a month, we sat down with the varied set of data available to us and talked about our sales performance and forecast. We learned from the data and from each other. We learned how to interpret the data, explained our data to other team members, and backed up our claims with evidence. We challenged each other and then outlined a robust, intelligent conclusion. I believe nothing can beat this approach because it mixes data - which is always imperfect - with knowledge and the bits and pieces of information at hand. All of this combined can help make informed decisions.”