Hadoop. How to avoid the hype. (with Tamara Dull of SAS)

Reading the Gartner Hype Cycle for 2015, one technology that was missing was big data. Betsy Burton, the Gartner analyst who put together the report explained that it is now such a part of many different aspects of our lives that it is within a myriad of new hype cycles. Take a look for yourselves here.

One of the interesting things that I have seen as a headhunter/recruitment consultant within among other areas, big data, is a whole host of companies charging out to deploy Hadoop and then their HR departments and internal recruiters frantically looking for "unicorns," to hire. What is interesting is how many of these deployments end in success and although I don't have the stats, I did watch an interesting presentation at Strata and Hadoop World, London 2015, regarding the success rate (or lack of) of the majority of Hadoop deployments out there. Here is a link to that presentation: Hadoop: As Easy As Riding a Bike.

In this talk, the speaker Tamara Dull explains how the success rates of Hadoop deployments are in the range of 25-33% and that two-thirds of the problems faced are typically down to people and not the technology in question. Tamara gives an analogy about riding a bike and cites the success rate of projects being about the rider and not the bike itself. I wanted to dig a little deeper, so I reached out to Tamara and am delighted to share her responses with you here.

A brief introduction about Tamara Dull is that she is the Director of Emerging Technologies for SAS Best Practices, a thought leadership team at SAS. Through provocative articles and publications, and key industry engagements, she delivers a pragmatic perspective on big data, the Internet of Things, open-source, privacy, and security. Tamara began her high-tech journey long before the internet was born, and has held both technical and management positions for multiple technology vendors, consultancies, and a non-profit. She is currently listed in IndustryWeek’s 25 Most Influential Women in IoT, the Big Data 2016: Top 100 Influencers and Brands list and The Top 100 Big Data Experts to Follow in 2016. You can also follow Tamara on Twitter by clicking here: Thanks so much for your participation Tamara.

Richard: "Hadoop is a major tech buzzword at the moment. What relevant questions should C-suite executives be asking themselves before deploying it and to avoid getting caught up in the hype?"

Tamara: "Here are three key questions executives should answer before getting started:

1. What business problem(s) do you want to solve? It's business first, technology second. Many companies get this turned around. Answer the "what" and "why" before you apply the "how."
2. Each big data initiative requires an executive sponsor. This sponsor may change from initiative to initiative. Will your initiative have an executive sponsor? If the answer is no, then it may become nothing more than an academic exercise.
3. Does your big data initiative tie back to a corporate strategy?Jill Dyche discusses the importance of this in her latest book, The New IT: Enabling Business Strategy in the Digital Age. Stay focused on what your company wants to achieve, and then get behind those projects and initiatives that will get you there. Abandon the rest."

Richard: "After watching your presentation at Strata and Hadoop World, London 2015, you indicated that the success rate of deploying Hadoop is "meh." Why is that and what are the common mistakes that you have seen within other organizations that can be avoided by others that follow?"

Tamara: "This ties back to my earlier response. Early on, many companies were failing because they weren't solving a "real" business problem and/or they didn't have executive support. So even if a "skunkworks" big data project succeeded, without executive support, it would die on the vine. Another common scenario has been IT claiming success (by installing Hadoop), and the business asking, "What am I supposed to do with this?" Let the business drive the need - i.e., define the problems to be solved, get executive sponsorship, etc. - and then IT can help with the technology."

Richard: "We recently heard that Hadoop would be an industry that tops $60BN by the end of the decade, growing annually at a rate of 26%. Why should C-suite executives be looking at it today and what other technologies when it comes down to big data should they be aware of and keeping tabs on?"

Tamara: "If we simplify what we want to do with data - collect/create, store, process/compute, and analyze - big data technologies like Hadoop do a very good job of storing and processing any and all types of data - at a fraction of the time and cost of our traditional, relational technologies. This has been an important nut for us to crack, given the volume of data that's now being generated. (Think the Internet of Things.) But the real value - and the bigger nut to crack - is those technologies that will help us identify and analyze the "right" data at the "right" time. "Right" is a relative term and differs for each company."

Richard: "When it comes down to big data strategy within an organization, what advice can you share with senior executives to give them the best possible chance of success?"

Tamara: "To be successful, I'd highlight two best practices:

• Make sure your big data strategy supports one or more corporate strategies. Spend your time and money on those strategies and initiatives that will get you to where you want and need to go as a company.
• Make sure the big data strategy is addressing a real business pain point. And then choose the right technology to tackle the problem. A big data problem will most likely be addressed by the company's existing technologies, along with the newer big data technologies like Hadoop or Spark. It doesn't have to be an either/or proposition."

Richard: "Finding skilled technical professionals within big data is a current and ongoing challenge. What impact do you feel this has had on organizations successfully deploying Hadoop?"

Tamara: "Getting these big data technologies, like Hadoop, set up, isn't the challenge. The more significant challenge is integrating these big data technologies into a company's existing technologies, processes, and people (culture) - yes, the golden trifecta of business transformation. Even if a company can find and hire one or more of these "unicorns," that won't solve the "people" challenge. This new era of big data, not to mention the Internet of Things, is a game changer for many companies. The companies that understand that big data is more than just a technology play will be the ones who succeed."

My name is Richard Downes, and I specialize in helping companies hire experienced Analytics, Data Science, Machine Learning and NLP professionals in Europe and the U.S and have over 15 years of Recruitment / Staffing experience. If you are considering your next career move or are a company needing to make an experienced hire within any of these areas, please feel free to get in touch and take a look at my video introduction embedded below.