BETA
This is a BETA experience. You may opt-out by clicking here

More From Forbes

Edit Story

Reducing The Cost Of Failure With DataOps

Forbes Technology Council

Chris Bergh is the founder, CEO, and head chef at DataKitchen. DataKitchen offers a complete Enterprise DataOps Platform.

When failure is expensive, managers avoid it at all costs. When there is an aversion to failure, team members play it safe, take fewer risks and cling to formulaic patterns.

Humans are hardwired to learn by iterating on a problem. Trial and error is an essential ingredient of transformative innovation. The key to agile innovation is to develop processes that reduce the cost of iterations. When failure has minimal consequences, employees are free to experiment. Ironically, giving employees the freedom to fail unlocks tremendous creativity.

Agile development reinforces a culture of iterative development, enabling data teams to continuously improve analytics using user feedback to guide priorities. This allows teams to fail inexpensively, bringing two significant benefits to analytics development:

1. Failure is a compelling teacher. Systematically eliminating errors brings a sense of urgency that can't be matched in driving team learning.

2. Iterating inexpensively on analytics enables delivery of value immediately, in short sprints. The data team usually knows what analytics the organization needs next week. It's tough to predict what analytics an enterprise will need in a year.

Creating A Process-Oriented Culture 

It's essential to cultivate a corporate culture that embraces errors, learns from them and utilizes them to guide priorities.

Some people might hear this and think this means writing a new mission statement, holding teambuilding events, hiring differently or restructuring the data organization. There may be a time and place for each of these steps, but I would argue it's more effective to create structured workflows using automation that enforce a controlled, repeatable, robust process for analytics creation and operations.

These methods hearken back to the work of quality pioneer William Edwards Deming, who advocated that robust and reliable factories depend on systemic processes, not heroic individuals. Likewise, a data analytics factory cannot rely solely on individual effort. In data organizations, innovation flows from highly optimized processes and workflows that enable the desired results. 

A manager, for example, can ask the data team to regression test analytics before deployment. A request like this will produce inconsistent results as individual behavior and ability tend to vary. Alternatively, a manager can lead the team to develop a regression test suite orchestrated to run on all analytics before deployment. The automated approach will yield consistent, robust and repeatable results. Furthermore, the test suite grows over time, improving its effectiveness. The process-centered approach works equally well at 4 a.m. as at 4 p.m. and equally well for junior or senior technical contributors.

Process-Centered Analytics With DataOps

Let's look at some concrete steps that data organizations can implement immediately to reduce the cost of failure. These include specific workflow automation and methods that minimize waste associated with data engineering and data science projects. These methods are collectively part of an approach to data analytics called DataOps, which takes a process-centric approach to streamline analytics workflows. 

DataOps aims at automating the workflows comprising the end-to-end data analytics lifecycle. It extends your existing toolchain using automation instead of manual methods. There is nothing special that you need to know about your existing data and tech stack before embarking on a DataOps initiative. DataOps methods can be applied to any existing enterprise architecture.

The DataOps enterprise software ecosystem consists of 100+ vendors. Many different tools can help with aspects of DataOps. You can't go wrong if you stay close to the objective of reducing the cost of failure. In concrete terms, that means minimizing the cycle time of workflows associated with the analytics lifecycle. Automating the following six aspects of your data lifecycle will deliver on these aims.

• Segregate analytics development from production — keep production safe and revision controlled.

• Automate the creation of on-demand development environments — stop spending months setting up environments for new projects.

• Perform impact review using an automated test suite – end bureaucratic impact review meetings.

• Align dev and production environments and parameterize environments so deployment occurs seamlessly.

• Test data and analytics at every stage of processing, gradually reducing errors to virtually zero.

• Create libraries of reusable components — stop reinventing the wheel and then supporting numerous versions of the wheel.

Processes That Reduce The Cost Of Failure

Streamlining the workflows related to analytics development enables data engineers and scientists to engage in trial and error much less expensively. DataOps automation removes cost from tasks such as creating environments, impact review, deployment and observability.

When these processes are optimized, analytics developers are more easily able to follow their curiosity and experiment. When the consequences of mistakes are minimized, developers can more freely iterate their way to success. 


Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?


Follow me on Twitter or LinkedInCheck out my website