The Big Data space has been confusing. Investors looking to ride the ups and downs of this market need to develop high tolerance for contradictory messages.
Take Hadoop for example. A few years ago, the technology was hailed as THE solution for all Big Data problems. Then, came the “IT research firm prediction wars”: Gartner claimed that only few companies had deployed it while Forrester forecasted that 100% of enterprises would have it by the end of next year.
Since then, two large companies servicing the Hadoop market have gone public with much financial appeal. Cloudera (NYSE:CLDR) went IPO in April and its valuation now nears $2.3B (the company is set to report earnings on September 9). Hortonworks (NASDAQ:HDP) reported last week that its stock has gone up by over 74% since January 1st. For reference, that’s more than Tesla (NASDAQ:TSLA) did in the same period!
So, what’s really happening here?
Hadoop expectations: the elephant in the room...
Hadoop has been around for about 10 years. And over the last 10 years, many innovations have been delivered, with technology breakthroughs that span a large space, from Apache Spark to Presto to Zeppelin.
Hadoop got it start as a batch system, primarily used to store large volumes of data at a low cost. Now, thousands of enterprises have delivered Hadoop-powered applications for financial fraud detection, cybersecurity, inventory management, network troubleshooting, risk analysis, IoT… etc. Some even claim to use Hadoop to save lives: Cerner, a Cloudera customer, says it has developed an infection detection system that has saved more than 3,000 lives to date!
So, let’s just say that next time you hear that “Hadoop has failed to live up to its promise”, refer back to the above few facts to put such comments back into perspective.
One more morsel of food for thought… If you don’t believe the value of this technology can be assessed effectively by looking at companies that only recently became public, consider that EMR, Amazon’s managed Hadoop framework, is part of a $12B business that grew 55% last year.
Bottom Line: No industry is perfect. No technology is without flaws. However, doomsday statements about Hadoop's failure in 2017 don't take into account how technology industry markets work.
How Do Technology Markets Work?!
For decades, observers looking to assess the maturity of an industry, have found help in models like Geoffrey Moore’s “Crossing The Chasm” curve to the Gartner Hype Cycle. Each model has its own issues of course, but there is one constant theme that runs through all of them: complete and immediate replatform rarely occurs. As new technologies emerge, companies adopt them complementarily to their current infrastructure.
Technology adoption is a process. If you’re skeptical about this fact, then ask CIOs why they still manage mainframe systems, a technology invented decades ago.
When it comes to Big Data, the enterprise CIO’s reality is not much different. Unless a company was ‘born with Hadoop’, every single enterprise CIO has to manage data stored in databases from the past (Oracle, SQL Server, Teradata, etc.), distributed platforms of the present (Hadoop) and new cloud serverless platforms (Google BigQuery). In short, the reality in the enterprise is that of mixed workloads, also known as hybrid. Enterprises have everything: Hadoop, non-Hadoop, in the cloud and on premises.
So, rather than saying “Hadoop has failed”, we should be asking which use-cases are most appropriate for which deployment mode: Hadoop, non-Hadoop, cloud, on-premise. We should also ask what their current stack is missing, collectively. Knowing we will have to manage a hodgepodge of technologies that run on different standards and constructs, we will need to ask, what are the consistent themes required to satisfy the growing needs of enterprise business employees?
Bottom Line: Big Data is here to stay. And Hadoop’s transformation power has been felt across thousands of organizations and millions of lives. Don’t believe the haters. The haters always hate. Stay curious, my friend. And, may you continue to trust in data.
=====
Bruno Aziza is a big data entrepreneur who focuses on growing businesses and turning them into global leaders. He has worked at startups, medium and large-size organizations globally, from Apple to BusinessObjects to Microsoft. Contact him at bruno.aziza@gmail.com and follow him on twitter @brunoaziza.
