Common pitfalls in delivering BIG DATA solutions.

History is littered with companies who have floundered, lost their way and failed.  Failed because they did not embrace change: Kodak, Motorola, Yahoo, Sun Microsystems... These organisations lost market share, haemorrhaged financially and entered a death spiral.  The lucky ones pulled out and came back fighting: Apple, Microsoft, IBM, Nokia…

Right now, we are in the midst of a digital revolution.  One that revolves around a tsunami of data. We all hear the buzz words: big data, the internet of things, business insights, predictive/descriptive/real-time analytics, smart data, machine learning, artificial intelligence, and we all hear about the rich rewards - after all, it’s a multi-billion-dollar industry!

So, no surprise, many companies are investing heavily in analytics and big data. Why then are many experiencing disappointing results and not seeing returns on their investments?

Often, there are organisational aspects.  Companies select the best technologies, but their organisation limits the chance of success.  But, there are other, more common, reasons. In this article, I will explore the first 4 of 8.

Pitfall 1.   Not Defining the Business Problem.

Let’s start with framing the business problem. Simple, right? 

Albert Einstein was reported as once saying, “If I were given one hour to save the planet, I would spend 59 minutes defining the problem and one minute resolving it.”

Despite the fact that this quote was likely not to have been said by Einstein, the sentiment remains: understanding the business problem is vital. Yet so many organisations fail to do this. They’re not rigorous enough and don’t spend sufficient time articulating the problem.

We’ve all seen the consequences: A project progressing down a particular path and then the ‘oh-no’ moment when it is clear they are solving a different problem, or worse still not solving any problem at all. Instead, they are building a solution that might then have to find a home!

When we define a problem correctly, it often reveals a simple and effective solution. However, the key to reaching this point is asking the right questions. “How?” some of you might ask. First, we must realise that problems can be symptomatic of deeper issues and quick fixes often solve surface issues and fail to tackle the root cause - sticking plasters rather than permanent solutions.

Many techniques can be used, but one (5 Whys) involves a deeper and more effective analysis. When you ask, the question “Why is this a problem?” the answer can often point to another problem. Thus, you move away from the initial problem to another problem. If you then ask, “Why is this a problem?” - and keep doing this - you eventually arrive at the source of all the problems.

Perhaps a quote by Benjamin Franklin is a good way to highlight this:

For the want of a nail, the shoe was lost.

For the want of a shoe, the horse was lost.

For the want of a horse, the rider was lost.

For the want of a rider, the battle was lost.

For the want of a battle, the kingdom was lost.

And all for the want of a nail.

Pitfall 2.   Getting Access to Data.

So, we’ve defined the business problem. Next, we need to gain access to the appropriate sources of data. Easy, right?

Sadly, this is not always the case. Legacy IT structures can hinder efforts and architectures prevent the integration of siloed information. Plus, managing unstructured data is often beyond traditional IT capabilities. Thus, fully resolving issues can take months, if not years. Yet, it is possible to achieve short-term and effective remedies by working with CIOs to identify and connect the most important data for the analytic problem. Whilst in parallel synchronising and merging overlapping data to focus on missing information.

Pitfall 3. Not Cleansing Data.

So, we’ve defined the business problem, we’ve gained access to the right sources of information. Easy sailing from here, right?

We are all familiar with the idiom ‘garbage in, garbage out’, yet too many organisations fail to clean and transform their data. Ironically, they then wonder why their analytic output and business insights fail to make sense.

Ensuring the data captured addresses the business problem is time-consuming but crucial. The best analytic methods and techniques won’t compensate for poor quality data.

Pitfall 4.   Not Choosing the Right Variables.

We've come this far - we've defined the business problem, gained access to the right data and we cleaned it. This is an achievement in itself..! So what could go wrong now?

It's back to understanding the business problem and defining response and explanatory variables. In short, clearly defining variables is key to having actionable insights.

I hear some of you ask - what is an explanatory variable and how does it differ from a response variable? In essence, the response variable is the focus of your business problem. Whereas, an explanatory variable (independent) is one that explains changes and includes anything that might affect the response variable.

Let’s say you’re trying to figure out whether Method 1 or Method 2 is better for business profitability. The question is which method will generate the better overall outcome? In this example cash generation, sales orders and downtime could all be response variables.

The method is the explanatory variable and may or may not affect the response variable. In this example, we have only one explanatory variable: the Method. In real life, you could have several more explanatory variables. (Read more here)

Your Experiences

So, has your organisation experienced any of these issues? Let me know in the comments section.

In Part 2 of this article, I will explore the other 4 common pitfalls.

Visualisation/Dashboards; Models/Rules; Refreshing Data and Alerts

Also, in future articles, we'll look at some real examples from different industries - from Finance to Retail to Oil & Gas.

Remember to follow me to read the next articles.