DataKitchen

AUGUST 8, 2024

Enjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.

IT 190

IT 190

DataKitchen

AUGUST 6, 2024

Chris Bergh joins me to chat about all things DataOps. We also discuss lean, removing waste from data processes and teams, and much more.

Data Processing 130

Data Processing 130

DataKitchen

AUGUST 6, 2024

Summary In this episode of the Data Engineering Podcast, host Tobias Macey welcomes back Chris Berg, CEO of DataKitchen, to discuss his ongoing mission to…

Data Processing 130

Data Processing 130

DataKitchen

AUGUST 6, 2024

DataKitchen’s Data Quality TestGen found 18 potential data quality issues in a few minutes (including install time) on data.boston.gov building permit data! Imagine a free tool that you can point at any dataset and find actionable data quality issues immediately! It sure beats having your data consumers tell you about problems they find when you are trying to enjoy your weekend.

Data Quality 124

Data Quality Testing Cost-Benefit Software 124

DataKitchen

JULY 8, 2024

Christopher Bergh, CEO of DataKitchen, is transforming data analytics with his DataOps approach. By applying principles from agile and lean manufacturing, Bergh aims to eliminate the 70-80% waste in data processes. DataKitchen's suite of open-source tools offers solutions for observability, testing, and automation, addresses challenges in rapid change management, error detection team productivity.

Manufacturing 130

Manufacturing Testing Data Processing Data Analytics 130

DataKitchen

JUNE 21, 2024

Navigating the Storm: How Data Engineering Teams Can Overcome a Data Quality Crisis Ah, the data quality crisis. It’s that moment when your carefully crafted data pipelines start spewing out numbers that make as much sense as a cat trying to bark. You know you’re in trouble when the finance team uses your reports as modern art installations rather than decision-making tools.

Data Quality 130

Data Quality Measurement Metrics Data Collection 130

DataKitchen

MAY 14, 2024

Data Observability and Data Quality Testing Certification Series We are excited to invite you to a free four-part webinar series that will elevate your understanding and skills in Data Observation and Data Quality Testing. This series is crafted for professionals eager to deepen their knowledge and enhance their data management practices, whether you are a seasoned data engineer, a data quality manager, or just passionate about data.

Data Quality 195

Data Quality Testing Metrics Measurement 195

DataKitchen

MAY 10, 2024

Harnessing Data Observability Across Five Key Use Cases The ability to monitor, validate, and ensure data accuracy across its lifecycle is not just a luxury—it’s a necessity. Data observability extends beyond simple anomaly checking, offering deep insights into data health, dependencies, and the performance of data-intensive applications. This blog post introduces five critical use cases for data observability, each pivotal in maintaining the integrity and usability of data throughout its journe

Data Quality 141

Data Quality Dashboards Strategy Testing 141

DataKitchen

MAY 10, 2024

The Five Use Cases in Data Observability: Ensuring Data Quality in New Data Sources (#1) Introduction to Data Evaluation in Data Observability Ensuring their quality and integrity before incorporating new data sources into production is paramount. Data evaluation serves as a safeguard, ensuring that only cleansed and reliable data makes its way into your systems, thus maintaining the overall health of your data ecosystem.

Data Quality 124

Data Quality Interactive Dashboards Testing 124

DataKitchen

MAY 10, 2024

The Five Use Cases in Data Observability: Effective Data Anomaly Monitoring (#2) Introduction Ensuring the accuracy and timeliness of data ingestion is a cornerstone for maintaining the integrity of data systems. Data ingestion monitoring, a critical aspect of Data Observability, plays a pivotal role by providing continuous updates and ensuring high-quality data feeds into your systems.

Data Quality 130

Data Quality Testing Software Dashboards 130

DataKitchen

MAY 10, 2024

The Five Use Cases in Data Observability: Mastering Data Production (#3) Introduction Managing the production phase of data analytics is a daunting challenge. Overseeing multi-tool, multi-dataset, and multi-hop data processes ensures high-quality outputs. This blog explores the third of five critical use cases for Data Observability and Quality Validation—data Production—highlighting how DataKitchen’s Open-Source Data Observability solutions empower organizations to manage this critical s

Metrics 124

Metrics Testing Data Quality Dashboards 124

DataKitchen

MAY 10, 2024

The Five Use Cases in Data Observability: Fast, Safe Development and Deployment (#4) Introduction The integrity and functionality of new code, tools, and configurations during the development and deployment stages are crucial. This blog post delves into the third critical use case for Data Observation and Data Quality Validation: development and Deployment.

Testing 124

Testing Cost-Benefit Data Quality Dashboards 124

DataKitchen

MAY 10, 2024

The post The Five Use Cases in Data Observability: Ensuring Accuracy in Data Migration first appeared on DataKitchen.

124

124

DataKitchen

MAY 7, 2024

DataKitchen Training And Certification Offerings For Individual contributors with a background in Data Analytics/Science/Engineering Overall Ideas and Principles of DataOps DataOps Cookbook (200 page book over 30,000 readers, free): DataOps Certificatio n (3 hours, online, free, signup online): DataOps Manifesto (over 30,000 signatures) One Day DataOps training (paid) Data Observability (the first step in DataOps) I deas and Principles of Data Observability Four-part Da

Data Quality 161

Data Quality Testing Consulting Metrics 161

DataKitchen

APRIL 18, 2024

Introducing DataKitchen’s Open Source Data Observability Software Today, we announce that we have open-sourced two complete, feature-rich products that solve the data observability problem: DataOps Observervability and DataOps TestGen. With these two products, you will know if your pipelines are running without error and on time and can finally trust your data.

Testing 117

Testing Data Quality Enterprise Data Analytics 117

DataKitchen

APRIL 11, 2024

The hosted by Christopher Bergh with Gil Benghiat from DataKitchen covered a comprehensive range of topics centered around improving the performance and efficiency of data teams through Agile and DataOps methodologies. Gil Benghiat, co-founder of Data Kitchen, began by explaining the overarching goal of achieving data team excellence, which involves delivering business value quickly and with high quality.

Statistics 152

Statistics Manufacturing Data Processing Data Quality 152

DataKitchen

MARCH 18, 2024

Today, Bristol Myers Squib (BMS) has fully acquired Karuna Therapeutics. We congratulate our customer on their fantastic success. Their product, KarXT, an antipsychotic, is revolutionary and is lined up for an FDA Prescription Drug User Fee Act (PDUFA) in September 2024 for the treatment of schizophrenia in adults. KarXT also has potential for the treatment of Alzheimer’s disease and Bipolar disorder.

Data Quality 23

Data Quality Dashboards Data Analytics Strategy 23

DataKitchen

MARCH 12, 2024

Key Success Metrics, Benefits, and Results for Data Observability Using DataKitchen Software Lowering Serious Production Errors Key Benefit Errors in production can come from many sources – poor data, problems in the production process, being late, or infrastructure problems. Reducing the errors your customers find and those they do not are key success metrics of Data Observability Using DataKitchen DataOps Observability and DataOps TestGen.

Metrics 117

Metrics Software Cost-Benefit Testing 117

DataKitchen

MARCH 5, 2024

ngx-toolkit, a new open-source project from DataKitchen At DataKitchen, we use Angular and strive for well-tested and maintainable code. We’ve created three libraries that have helped accelerate Angular development in our software projects. We are proud today to present to the open source community our monorepo with some of the libraries we have developed for Angular and Jest.

Testing 141

Testing Software IT 141

DataKitchen

MARCH 5, 2024

Why Not Hearing About Data Errors Should Worry Your Data Team In the chaotic lives of data & analytics teams, a day without hearing of any data-related errors is a blessing. Your team is on top of things, deliveries are on schedule (you think), and no major complaints are making their way to your desk. It’s tempting to adopt the “ What, me worry?

Data Quality 146

Data Quality Measurement Testing Data-driven 146

DataKitchen

FEBRUARY 27, 2024

Your LLM Needs a Data Journey: A Comprehensive Guide for Data Engineers The rise of Large Language Models (LLMs) such as GPT-4 marks a transformative era in artificial intelligence, heralding new possibilities and challenges in equal measure. LLMs have the potential to revolutionize how we interact with data, automate processes, and extract insights.

Data Quality 117

Data Quality Unstructured Data Testing Interactive 117

DataKitchen

FEBRUARY 23, 2024

DataKitchen Resource Guide To Data Journeys & Data Observability & DataOps Data (and Analytic) Observability & Data Journey – Ideas and Background Data Journey Manifesto and Why the Data Journey Manifesto? Five Pillars of Data Journeys Data Journey First DataOps “You Complete Me,” said Data Lineage to Data Journeys. Bridging the Gap: How ‘Data in Place’ and ‘Data in Use’ Define Complete Data Observability The Need For Personalized Data Journeys for Your Data Consumers Data Te

Testing 117

Testing Data-driven Metrics Data Quality 117

DataKitchen

FEBRUARY 22, 2024

The Art of Data Buck-Passing 101: Mastering the Blame Game in Data and Analytic Teams Welcome, dear readers, to the hallowed halls of Data Buck-Passing University, where the motto is “ Per Alios Culpa Transfertur ” (Blame is Transferred to Others). In the world of data and analytics, one skill stands timeless and universal: the art of blaming someone else when things go sideways.

Analytics 183

Analytics Data Governance Strategy Testing 183

DataKitchen

DECEMBER 7, 2023

Do you have data quality issues, a complex technical environment, and a lack of visibility into production systems? These challenges lead to poor quality analytics and frustrated end users. Getting your data reliable is a start, but many other problems arise even if your data could be better. And your customers don't care where the problem is in your toolchain.

Uncertainty 130

Uncertainty Data Quality Dashboards Analytics 130

DataKitchen

NOVEMBER 10, 2023

The Perilous State of Today’s Data Environments Data teams often navigate a labyrinth of chaos within their databases. The core issue plaguing many organizations is the presence of out-of-control databases or data lakes characterized by: Unrestrained Data Changes: Numerous users and tools incessantly alter data, leading to a tumultuous environment. Extrinsic Control Deficit: Many of these changes stem from tools and processes beyond the immediate control of the data team.

Data Quality 136

Data Quality Testing Data Lake Data Integration 136

DataKitchen

NOVEMBER 6, 2023

This webinar discusses how to make embarrassing data errors a thing of the past. We will start with how data engineers do not understand their data and have difficulty identifying problematic data records. We will also discuss how the vast majority of data engineers are so busy that they don’t know, or have time to write, tests to write to find data errors.

Testing 130

Testing IT 130

DataKitchen

OCTOBER 20, 2023

In today’s data-driven landscape, Data and Analytics Teams i ncreasingly face a unique set of challenges presented by Demanding Data Consumers who require a personalized level of Data Observability. As opposed to receiving one-size-fits-all status updates, these key stakeholders desire real-time, granular insights into the status of their specific data as it traverses the complicated data production pipeline.

Insurance 169

Insurance Metadata Data-driven Data Quality 169

DataKitchen

OCTOBER 12, 2023

The meeting room was spacious, adorned with charts, graphs, and several cups of hastily grabbed coffee. I distinctly remember an afternoon set to work with a promising prospect. As my team and I waited, a flurry of messages informed us that the key players from the other side wouldn’t be joining. Why? Their entire data analytics team had been hastily yanked into an impromptu “war room” to tackle an emergency data issue.

Testing 117

Testing Data-driven Data Analytics Reporting 117

DataKitchen

OCTOBER 2, 2023

Data Teams and Their Types of Data Journeys In the rapidly evolving landscape of data management and analytics, data teams face various challenges ranging from data ingestion to end-to-end observability. This comprehensive article delves into the complexities encountered by various types of data teams—Data Ingestion Teams, End-to-End Data Product Teams, and Enterprise Data Enablement Teams—to name a few.

Data Quality 100

Data Quality Testing Uncertainty Data Enablement 100

DataKitchen

SEPTEMBER 21, 2023

Bridging the Gap: How ‘Data in Place’ and ‘Data in Use’ Define Complete Data Observability In a world where 97% of data engineers report burnout and crisis mode seems to be the default setting for data teams, a Zen-like calm feels like an unattainable dream. Complaints from dissatisfied customers and apathetic data providers only add to the mounting stress.

Testing 169

Testing Data Quality Predictive Modeling Metrics 169

DataKitchen

SEPTEMBER 20, 2023

This webinar discusses how to make embarrassing data errors a thing of the past. We will start with how data engineers do not understand their data and have difficulty identifying problematic data records. We will also discuss how the vast majority of data engineers are so busy that they don’t know, or have time to write, tests to write to find data errors.

Testing 130

Testing IT 130

DataKitchen

AUGUST 24, 2023

Since the first edition of the DataOps Cookbook in 2019, we have talked with thousands of companies about their struggles to deliver data-driven insight to their customers. In many ways, they all have the same problems. They have built data and analytic systems with great hope of success. Still, they are burdened with too many errors, are overwhelmed with custom requests, and know they fail to succeed in their goal of leading their organizations to be more data-driven.

Data-driven 130

Data-driven Forecasting Metrics Dashboards 130

DataKitchen

AUGUST 8, 2023

On 20 July 2023, Gartner released the article “ Innovation Insight: Data Observability Enables Proactive Data Quality ” by Melody Chien. In the article, Melody Chien notes that Data Observability is a practice that extends beyond traditional monitoring and detection, providing robust, integrated visibility over data and data landscapes. It alerts data and analytics leaders to issues with their data before they multiply.

Data Quality 130

Data Quality Testing Snapshot Reporting 130

DataKitchen

JULY 27, 2023

The Ten Standard Tools To Develop Data Pipelines In Microsoft Azure. While working in Azure with our customers, we have noticed several standard Azure tools people use to develop data pipelines and ETL or ELT processes. We counted ten ‘standard’ ways to transform and set up batch data pipelines in Microsoft Azure. Is it overkill? Don’t they all do the same thing?

Machine Learning 130

Machine Learning Cost-Benefit Data Transformation Testing 130

DataKitchen

JULY 12, 2023

The Syntax, Semantics, and Pragmatics Gap in Data Quality Validate Testing Data Teams often have too many things on their ‘to-do’ list. Customers are asking for new data, people need questions answered, and the tech stack is barely running – data engineers don’t have time to create tests. They have a backlog full of new customer features or data requests, and they go to work every day knowing that they won’t and can’t meet customer expectations.

Data Quality 130

Data Quality Testing Manufacturing Finance 130

DataKitchen

JULY 3, 2023

Data Journey First DataOps Putting Problems in Your Data Estate at the Forefront Welcome to the high-octane world of DataOps, a powerhouse that turbocharges data analytics development and management. This innovative approach merges the agility of Agile Development, the stability of DevOps, and the meticulousness of Statistical Process Controls, orchestrating a dynamic, enriched, and nimble data ecosystem that is truly remarkable.

Testing 130

Testing Risk Data-driven Statistics 130

DataKitchen

JUNE 19, 2023

DataOps TestGen Technical Product Overview

130

130