BETA
This is a BETA experience. You may opt-out by clicking here

More From Forbes

Edit Story

DataOps Lowers The Cost Of Asking Analytic Questions

Forbes Technology Council

Chris Bergh is the founder, CEO, and head chef at DataKitchen. DataKitchen offers a complete Enterprise DataOps Platform.

How much does it cost to ask your data or business analytics team a question? This metric could be the key to winning the next business cycle.

The data team provides an enterprise with analytic insights that serve as the foundation for operations, revenue acquisition and data-driven decision-making. Every day, co-workers pose new questions to the data team. These requests require new analytics, updated charts and graphs, new models, dashboard changes and integrations of new data sets.

Most data teams can't keep up with the endless stream of new requests. Business users often wait so long for new analytics that the utility of a particular request expires before it is addressed. Executives are forced to make decisions without the analytics that they need.

Analytics teams tend to be very unanalytical about themselves. Perhaps this is because the work they do is performed by highly capable specialists. Those who don't understand the delicate art of machine learning decide it's best to leave it up to the data scientists. If we take a bird's-eye view of data analytics, it's possible to optimize analytics workflows using management methods that have been successful in other domains.

Cost Per Question

It helps to think of data analytics as a factory. Consider all of the fixed and variable costs related to data factory operations. This includes technology, human resources and overhead — the total operating cost of data and analytics.

Now, consider the total output of the data team — all of the charts, graphs and models they create in response to user stories and requests from business units. Note that bug fixes and unplanned work performed to resolve outages don't count in total output. These activities are critical, but they are, in effect, the carrying cost of technical debt. Resolving problems does nothing to answer the latest business question, which must be the data team's number one priority. All data team responses to requests from business stakeholders and other internal customers represent the value-add output of the data and analytics function.

The total output of the data team divided by the total cost of data analytics could be conceptualized as a metric called the "cost per question."

If you find yourself waiting too long for the data team to address requests, then the cost per question is likely too high. Behind the scenes, the data team is busy ordering new machines, purchasing new software licenses, working with a third-party data supplier, shepherding proposed changes through a bureaucratic impact review process or interfacing with a centralized IT team to get access to data or make a change to mastered data. Data delays prevent the business unit from receiving the analytics that they need in order to pursue opportunities or respond to competitive threats.

If an enterprise can lower the cost per question, they can ask a lot more questions. They can ask nuanced questions that uncover latent customer needs or underlying market dynamics. Lowering the cost per question of analytics can boost business agility, spurring innovation.

How DataOps Lowers Cost Per Question

An organization can lower the cost per question using the same proven techniques that manufacturers use to reduce costs in other factories. Raw data comes into the data factory and progresses through a series of transformational steps until it is eventually published as charts and graphs. Finished analytics are the finished goods of the data factory. To lower costs, a manufacturing company applies methods such as lean manufacturing to eliminate waste. In a data factory, lean manufacturing requires the data team to adopt a process-oriented approach to their activities. It calls for DataOps across the end-to-end data life cycle.

DataOps automates data-related workflows so that data professionals are freed from performing slow and error-prone manual steps. It automates data operations, the creation of on-demand development environments, the deployment of new analytics to production and necessary controls like governance and monitoring. DataOps also tests data and analytics at every step of processing so that errors are gradually eliminated.

These strategies lower the cost of technical debt so that it stops interfering with productivity. The higher quality ensured by DataOps reduces unplanned work, enabling the data team to stay focused on user requests.

How To Get Started With DataOps

DataOps can and should be implemented in small steps that complement and build upon existing workflows and data pipelines. To get started with DataOps, work with an internal or external expert to add testing and monitoring to your existing production pipelines to create a highly observable, error-free analytic factory of insight. Continuous testing, monitoring and observability can be integrated into your existing data pipelines with little impact on existing processes.

The next phase should address development bottlenecks. Expand DataOps to your development workflows to accelerate analytic cycle time and reduce deployment risk. DataOps capabilities such as environment creation, continuous deployment, meta-orchestration and collaboration produce significant improvements in developer productivity with a small amount of process impact.

Subsequent DataOps investment can add process transparency to establish metrics that demonstrate improvement and shine a light on areas that can benefit from further DataOps investment. DataOps process metrics help the data team understand activity at an aggregate level with an ability to drill down to the lowest level of detail. Metrics and dashboards can be useful touchstones as you expand DataOps to multiple organizations within the enterprise.

As data gradually becomes more strategically important, companies with the lowest cost per analytics question will gain a competitive advantage over their less innovative peers. Industry-leading data analytics capabilities will drive more accurate and agile data-driven decision-making. Lean processes for extracting value from data will emerge as critical core competencies that drive innovation and business agility.


Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?


Follow me on Twitter or LinkedInCheck out my website