DataRobot Blog

MAY 26, 2022

To start with, SR 11-7 lays out the criticality of model validation in an effective model risk management practice: Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses.

Risk 52

Risk Modeling Metrics Business Objectives 52

O'Reilly on Data

JULY 28, 2020

Product Managers are responsible for the successful development, testing, release, and adoption of a product, and for leading the team that implements those milestones. When a measure becomes a target, it ceases to be a good measure ( Goodhart’s Law ). The Core Responsibilities of the AI Product Manager.

Marketing 363

Marketing Experimentation Metrics Testing 363

Rocket-Powered Data Science

FEBRUARY 15, 2023

3) How do we get started, when, who will be involved, and what are the targeted benefits, results, outcomes, and consequences (including risks)? Keep it agile, with short design, develop, test, release, and feedback cycles: keep it lean, and build on incremental changes. Test early and often. Test and refine the chatbot.

Strategy 290

Strategy Experimentation Uncertainty Machine Learning 290

DataKitchen

NOVEMBER 18, 2022

As he thinks through the various journeys that data take in his company, Jason sees that his dashboard idea would require extracting or testing for events along the way. So, the only way for a data journey to truly observe what’s happening is to get his tools and pipelines to auto-report events. Data and tool tests.

Testing 130

Testing Statistics Measurement Metrics 130

DataKitchen

JUNE 19, 2023

Another way to look at the five pillars is to see them in the context of a typical complex data estate. Monitoring is another pillar of Data Journeys, extending down the stack. Checking data at rest involves looking at syntactic attributes such as freshness, distribution, volume, schema, and lineage.

Testing 130

Testing Cost-Benefit Metrics Data Quality 130

DataKitchen

MARCH 16, 2023

How to measure your data analytics team? So it’s Monday, and you lead a data analytics team of perhaps 30 people. Like most leaders of data analytic teams, you have been doing very little to quantify your team’s success. The Active Data Ratio metric determines the percentage of datasets that deliver value.

Metrics 130

Metrics Data Analytics Analytics Measurement 130

IBM Big Data Hub

NOVEMBER 29, 2023

They are already identifying and exploring several real-life use cases for synthetic data, such as: Generating synthetic tabular data to increase sample size and edge cases. You can combine this data with real datasets to improve AI model training and predictive accuracy. This is partially true for synthetic data.

Metrics 80

Metrics Machine Learning Statistics Risk 80

CIO Business Intelligence

JUNE 9, 2023

It’s a time-tested truth: Getting a head start improves outcomes. A few years ago, Gartner found that “organizations estimate the average cost of poor data quality at $12.8 million per year.’” Beyond lost revenue, data quality issues can also result in wasted resources and a damaged reputation.

Strategy 105

Strategy Management Key Performance Indicator ROI 105

O'Reilly on Data

DECEMBER 12, 2019

1] This includes C-suite executives, front-line data scientists, and risk, legal, and compliance personnel. These recommendations are based on our experience, both as a data scientist and as a lawyer, focused on managing the risks of deploying ML. 6] Debugging may focus on a variety of failure modes (i.e.,

Machine Learning 198

Machine Learning Modeling Testing Risk Management 198

CIO Business Intelligence

NOVEMBER 13, 2023

Transformational leaders must ensure their organizations have the expertise to integrate new technologies effectively and the follow-through to test and troubleshoot thoroughly before going live. Ensuring data quality, privacy, and security is essential.

Digital Transformation 135

Digital Transformation Cost-Benefit Experimentation Measurement 135

CIO Business Intelligence

MARCH 24, 2023

The Business Application Research Center (BARC) warns that data governance is a highly complex, ongoing program, not a “big bang initiative,” and it runs the risk of participants losing trust and interest over time.

Data Governance 142

Data Governance Management Metadata Data Quality 142

datapine

JANUARY 24, 2021

This can include a multitude of processes, like data profiling, data quality management, or data cleaning, but we will focus on tips and questions to ask when analyzing data to gain the most cost-effective solution for an effective business strategy. 3) Where will your data come from?

IT 317

IT Statistics KPI Data-driven 317

Anmut

APRIL 6, 2023

What are the Benefits of Doing a Data Maturity Assessment? How do you do a Data Maturity Assessment? What does a Data Maturity Assessment Measure? What makes a Good Data Maturity Assessment? What is a Data Maturity Assessment? What does a Data Maturity Assessment Measure?

Cost-Benefit 52

Cost-Benefit Data Quality Data Governance Measurement 52

O'Reilly on Data

OCTOBER 22, 2020

It’s also a good indirect measure of training data quality: a team that does not know where their data originated is likely to not know other important details about the data as well. ML models directly involved in critical safety systems are generally easy to identify as areas of concern.

Consulting 303

Consulting Risk Cost-Benefit Modeling 303

datapine

APRIL 15, 2020

Your Chance: Want to test an agile business intelligence solution? Business intelligence is moving away from the traditional engineering model: analysis, design, construction, testing, and implementation. Test BI in a small group and deploy the software internally. Finalize testing. Without further ado, let’s begin.

Business Intelligence 276

Business Intelligence Analytics Testing Dashboards 276

CIO Business Intelligence

MARCH 7, 2024

Public LLMs in the sandbox: Safely test publicly available Large Language Models (LLMs) in a sandbox environment, separate from the production setting, to assess their impact without risking operational disruptions. Many businesses are in the exploratory phases, creating unique business cases and testing their theory on uses.

Digital Transformation 72

Digital Transformation Machine Learning Risk Testing 72

erwin

JANUARY 11, 2019

It’s only when companies take their first stab at manually cataloging and documenting operational systems, processes and the associated data, both at rest and in motion, that they realize how time-consuming the entire data prepping and mapping effort is, and why that work is sure to be compounded by human error and data quality issues.

Data Governance 40

Data Governance Risk Metadata Management 40

AWS Big Data

AUGUST 14, 2023

At the same time, unstructured approaches to data mesh management that don’t have a vision for what types of products should exist and how to ensure they are developed are at high risk of creating the same effect through simple neglect. When do new data products get created, and who is allowed to create them?

Management 78

Management Risk Measurement Optimization 78

DataRobot

JULY 16, 2021

They all serve to answer the question, “How well can my model make predictions based on data?” In performance, the trust dimensions are the following: Data quality — the performance of any machine learning model is intimately tied to the data it was trained on and validated against. Operations.

Machine Learning 98

Machine Learning Uncertainty Modeling Measurement 98

erwin

APRIL 15, 2020

If we reflect on the coronavirus example, we can ask: How do individual countries report their data? Do individual countries use the same testing protocols? Are infections universally defined the same way (based on widely administered tests or only hospital admissions)? Data Governance Methods for Data Distancing.

Data Governance 134

Data Governance Metadata Testing Reporting 134

DataRobot

JUNE 1, 2021

In an earlier post, I shared the four foundations of trusted performance in AI : data quality, accuracy, robustness and stability, and speed. You should first identify potential compliance risks, with each additional step again tested against risks. We think of this as the operational environment of an AI system.

Uncertainty 52

Uncertainty Machine Learning Advertising Risk 52

Alation

MARCH 22, 2022

As the latest iteration in this pursuit of high-quality data sharing, DataOps combines a range of disciplines. It synthesizes all we’ve learned about agile, data quality , and ETL/ELT. So, how can you quickly take advantage of the DataOps opportunity while avoiding the risk and costs of DIY?

Metadata 52

Metadata Cost-Benefit Data Quality Data Lake 52

O'Reilly on Data

MARCH 31, 2020

The model outputs produced by the same code will vary with changes to things like the size of the training data (number of labeled examples), network training parameters, and training run time. This has serious implications for software testing, versioning, deployment, and other core development processes.

Management 339

Management Machine Learning Experimentation Metrics 339

CIO Business Intelligence

SEPTEMBER 2, 2022

The average salary for a financial software engineer is $116,670 per year, with a reported salary range of $85,000 to $177,000 per year, according to data from Glassdoor. Full-stack software engineers are essentially high-level software engineers who are focused on designing, testing, and implementing software applications.

Finance 98

Finance IT Software Reporting 98

CIO Business Intelligence

AUGUST 31, 2022

The average salary for a financial software engineer is $116,670 per year, with a reported salary range of $85,000 to $177,000 per year, according to data from Glassdoor. Full-stack software engineers are essentially high-level software engineers who are focused on designing, testing, and implementing software applications.

Finance 98

Finance IT Software Reporting 98

Cloudera

JUNE 29, 2023

To ensure the stability of the US financial system, the implementation of advanced liquidity risk models and stress testing using (MI/AI) could potentially serve as a protective measure. To improve the way they model and manage risk, institutions must modernize their data management and data governance practices.

Data Architecture 85

Data Architecture Risk Management Risk Management 85

DataKitchen

SEPTEMBER 21, 2023

In the context of Data in Place, validating data quality automatically with Business Domain Tests is imperative for ensuring the trustworthiness of your data assets. Running these automated tests as part of your DataOps and Data Observability strategy allows for early detection of discrepancies or errors.

Testing 169

Testing Data Quality Predictive Modeling Metrics 169

O'Reilly on Data

JULY 15, 2019

In recent posts, we described requisite foundational technologies needed to sustain machine learning practices within organizations, and specialized tools for model development, model governance, and model operations/testing/monitoring. Note that the emphasis of SR 11-7 is on risk management.). Sources of model risk.

Machine Learning 38

Machine Learning Management Enterprise Risk Management 38

Domino Data Lab

JUNE 23, 2019

As a result, Skomoroch advocates getting “designers and data scientists, machine learning folks together and using real data and prototyping and testing” as quickly as possible. These measurement-obsessed companies have an advantage when it comes to AI. It is similar to R&D. Transcript. Hi, I’m Peter Skomoroch.

Management 110

Management Machine Learning Experimentation Metrics 110

O'Reilly on Data

JUNE 19, 2019

More precisely, models built or tuned for specific applications (in reality, this means models + data ) will need to be managed and protected : A database for authorization and security: who has read/write access to certain models. A catalog or a database that lists models, including when they were tested, trained, and deployed.

Modeling 196

Modeling Machine Learning Testing Metrics 196

CIO Business Intelligence

MARCH 14, 2024

Adopting and scaling AI Some of the NAO’s advice to government could also apply to enterprises, especially those with a similar aversion to risk and change. Testing and piloting AI is key to successful adoption, the study found, and the government bodies surveyed are doing well on this score, with 70% already doing so.

Testing 101

Testing Risk Digital Transformation Statistics 101

Cloudera

SEPTEMBER 7, 2023

But it’s also fraught with risk. This June, for example, the European Union (EU) passed the world’s first regulatory framework for AI, the AI Act , which categorizes AI applications into “banned practices,” “high-risk systems,” and “other AI systems,” with stringent assessment requirements for “high-risk” AI systems.

Insurance 83

Insurance Risk Data-driven Finance 83

DataRobot Blog

JULY 21, 2022

Inquire whether there is sufficient data to support machine learning. Outline clear metrics to measure success. Document assumptions and risks to develop a risk management strategy. Define the exact calculation for the target variable or create a couple options to test. Test for bias to ensure fairness.

Machine Learning 98

Machine Learning Metrics Modeling Testing 98

Andrew White

JANUARY 11, 2021

But we also know not all data is equal, and not all data is equally valuable. Some data is more a risk than valuable. Additionally, the value of data may change, and our own personal judgement of the the same data and its value may differ. Risk Management (most likely within context of governance).

Data Analytics 52

Data Analytics Analytics Data-driven Finance 52

Alation

OCTOBER 28, 2021

Leaders are asking how they might use data to drive smarter decision making to support this new model and improve medical treatments that lead to better outcomes. Yet this is not without risks. To make good on this potential, healthcare organizations need to understand their data and how they can use it.

Data Governance 52

Data Governance Measurement Metrics Data Quality 52

datapine

AUGUST 29, 2022

The book has three main ideas: The biggest risk your company faces is investing a lot of time and resources into building something that the market doesn’t want. and this book will give you an insight into their data collecting procedures and the reasons behind them. Product/market fit is THE most important factor to get right.

Big Data 263

Big Data Data Analytics Analytics Data mining 263

datapine

NOVEMBER 27, 2019

Companies are no longer wondering if data visualizations improve analyses but what is the best way to tell each data-story. 2020 will be the year of data quality management and data discovery: clean and secure data combined with a simple and powerful presentation. 1) Data Quality Management (DQM).

Business Intelligence 361

Business Intelligence Analytics Prescriptive Analytics Data Quality 361