Most often, one hears remarks that Big Data implementation is a failure. This requires a reset of expectations. Big Data is all hype. With this post, I would like to share my views.

In the last 10 years, I’ve spent 5 ½ years in the consulting space and the remaining years with end user clients. By no means do I consider myself an expert; however, I would like to elaborate what I learnt through out these years across both sides of the fence.

Let us understand why do Big Data projects fail. Before doing that, let us agree on the definition of a failure. In my views, a project or an implementation is considered a failure if it does not deliver the perceived benefits.

Most Big Data projects fail because there is very little or no business initiatives and no proper use cases identified. These projects usually run as a technical proposal or the CIO’s initiative. If one looks at the data value chain, the most important part of the chain is Data Operationalisation and/or Data Monetisation. If a project runs as a technical initiative, the Data Monetisation part is often ignored and hence, there is no ROI on the investments.

Most of the Consulting Companies sell Big Data solutions as a silver bullet solution to all the enterprises’ problems and this creates lots of hype. When hype meets reality, the outcome is often not pleasant.

An ideal approach to this should be as:

Client - “I want to implement a Big Data solution”

Consultant- “That could be the solution, but let us discuss the problem first”.

Having been in the consulting space for 5 ½ years, I can say from experience that once you establish yourself as a trusted advisor, the business is only going to grow. From an end-user’s perspective my advice is to use a common-sense approach. If something is too good to be true then probably it is not true. This will help both parties to differentiate between hype and reality.

A few other useful pieces of advice from experience:

• Implement Big Data Solution only if you have proper use cases. Don’t do this because everyone else is doing this.
• Be conservative in your investments- Crawl before you walk and walk before you run. Use the elasticity feature in public clouds, if you can. Start small with agreed use cases, show return on investments before investing any more money. The lesser the cost the lesser the pressure to show the benefits.
• Involvement of business is very critical for proper adoption and benefit(s) realisation. Data monetisation is the key part to recover your investments and that cannot happen without business involvement.
• Invest in your resources, these are key to the system adoption.
• Last but not least, manage the change carefully, as this is a huge shift from RDBMS based systems to the ungoverned world of raw data.

Harjot Singh has over 22 years of experience in Data and Digital Transformation in Asia, Pacific, Africa and Middle East. He is currently working as SVP/ Head of Data Engineering for a Digital Banking Fintech in Dubai.