BETA
This is a BETA experience. You may opt-out by clicking here

More From Forbes

Edit Story

Managing Data Analytics Is More Like Running A Restaurant Than You Think

Forbes Technology Council

Chris Bergh is the founder, CEO, and head chef at DataKitchen. DataKitchen offers a complete Enterprise DataOps Platform.

Over the past few years, I’ve devoted significant energy to speaking and writing about a data analytics methodology called DataOps, which recognizes analytics systems as manufacturing processes. The data factory transforms raw materials (data) into finished goods (analytics) using a series of processing steps. As such, applying manufacturing methods, such as lean manufacturing, to data analytics produces tremendous results. Let’s explore this further using an analogy.

Imagine a restaurateur owns several fine dining restaurants that experience quality problems. Four out of 10 plates that her restaurants prepare are wonderful, but the other six have a problem. Maybe they are late. Perhaps they are prepared incorrectly, or maybe they fail to incorporate the customer’s special request. At this rate of failure, our intrepid entrepreneur wonders if her business will be able to survive. She also worries about the cycle time of innovation. Her chefs have many exciting new ideas, yet it takes so long to set up a new menu item that innovation languishes. What can be done? She hires a high-priced consultant.

The consultant studies her operations and recommends investing in “big food” – great new ovens that cook food in parallel. She tries that, and it doesn’t solve her problems. She hires a different consultant who interviews her entire staff. They recommend that she hire food scientists to create new dishes using “Asian-influenced food” (a.k.a. AI). She implements that suggestion, and their new dishes are great, but they still take too long to develop, and 60% of her plates continue to get sent back to the kitchen. 

Our restaurateur decides to fire the high-priced consultants and buys a book about lean manufacturing. It encourages her to study her errors. She quickly sees that her problem is not with her people or the tools they are using. Her issues stem from workflow methodologies. 

Focusing on the “wrong ingredients” problem, she instructs her team to attach a checklist to each plate. Before the plate goes out, the staff should make sure it has the proper ingredients, including special customer requests. Suddenly, the number of plate errors drops from six to four out of every 10. Our hero feels encouraged and continues to look for additional sources of errors.

She discovers that most of the problems are in plates served with chicken. She investigates the supply chain and finds that her warehouse is the last stop in her chicken supplier’s delivery route. Her staff begins to measure the temperature of arriving chicken, rejecting any meat that doesn’t conform to safety requirements. 

Our restaurant owner starts to look at the flow of work — from sous-chefs to chefs to plating to waitstaff to bussers — and discovers many bottlenecks and opportunities to find and resolve problems. The restaurant’s plate errors drop to virtually zero.

With far fewer errors to correct, our CEO has a lot more time to meet with her chefs and bring their innovations to the customers. She can plan out new workflows, create bills of materials, issue work instructions and safety measures. She devotes considerably more time to bringing new dishes to the menu — her true passion.

Lessons From The Kitchen

As illustrated in our example, buying new tools or hiring skilled experts will not address data analytics process and workflow problems. Data organization leaders need to take a page from manufacturing and apply the process-centric tools of Deming, the Toyota Way and other quality methods. Data teams can reduce their errors to virtually zero, minimize work in progress and slash development cycle time by applying proven methodologies like lean manufacturing.

Agile Development

The software industry uses a methodology called Agile to break large development projects into short components and release them in rapid, successive iterations. Agile monetizes features more quickly and avoids nonvalue-added work by relying upon user feedback to guide development priorities. Agile can be understood as an application of the Theory of Constraints (“The Goal”) to analytics development.

DevOps

Many enterprises have achieved greater agility through DevOps, automating development environment creation, testing/verification and release deployment. Inspired by lean manufacturing, DevOps slashes analytics development cycle time and improves quality while minimizing wasted effort. In data analytics, DevOps automation enables data teams to respond to requests for new analytics in hours or days, where manual methods previously took weeks or months. 

Quality

Enterprises now have trillions of discrete data values flowing through their systems. There’s no way to keep up without adding observability to the data architecture. Automated testing validates raw data and work in progress at each step of data transformation. When data falls outside the parameters of business logic, statistical probability or validity, automated tests can flag or trap the problem before it wreaks havoc. Continuous quality improvement in data analytics can reduce data and analytics errors to virtually zero while providing fine-grained transparency into workflows and data operations. Testing the data analytics pipelines is analogous to W. Edwards Deming’s pioneering work in manufacturing statistical process control

The data industry uses the term DataOps when applying these powerful methods to data analytics. DataOps is not a single vendor or tool. It’s a methodology that focuses on improving cycle time, quality and analytics agility using process automation. You can buy purpose-built tools that produce DataOps benefits, but a holistic approach to your end-to-end data life cycle has the best chance of unifying your toolchains and delivering impactful change.

Data teams are just beginning to understand how to manage analytics workflows from a process perspective. Those who understand the philosophy behind DataOps and embrace it as a methodology can get a jump on competitors. DataOps promises to do for data analytics what lean manufacturing and other quality methods did for manufacturing enterprises. The analytics teams that implement DataOps first will drive their organizations to make faster and more trustworthy data-driven decisions. With DataOps, data-analytics agility will become a core competency that builds and sustains market leadership.


Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?


Follow me on Twitter or LinkedInCheck out my website