O'Reilly on Data

OCTOBER 13, 2020

Ideally, AI PMs would steer development teams to incorporate I/O validation into the initial build of the production system, along with the instrumentation needed to monitor model accuracy and other technical performance metrics. But in practice, it is common for model I/O validation steps to be added later, when scaling an AI product.

Management 240

Management Metrics Machine Learning Modeling 240

The Unofficial Google Data Science Blog

JULY 18, 2023

Once we’ve answered that, we will then define and use metrics to understand the quality of human-labeled data, along with a measurement framework that we call Cross-replication Reliability or xRR. Last, we’ll provide a case study of how xRR can be used to measure improvements in a data-labeling platform.

Measurement 52

Measurement Metrics Uncertainty Slice and Dice 52

Domino Data Lab

JUNE 2, 2019

Co-chair Paco Nathan provides highlights of Rev 2 , a data science leaders summit. We held Rev 2 May 23-24 in NYC, as the place where “data science leaders and their teams come to learn from each other.” If you lead a data science team/org, DM me and I’ll send you an invite to data-head.slack.com ”.

Data-driven 110

Data-driven Data Science Machine Learning Modeling 110

The Unofficial Google Data Science Blog

AUGUST 27, 2019

Of course, any mistakes by the reviewers would propagate to the accuracy of the metrics, and the metrics calculation should take into account human errors. If we could separate bad videos from good videos perfectly, we could simply calculate the metrics directly without sampling. The missing verdicts create two problems.

Metrics 98

Metrics Statistics Uncertainty Optimization 98

Insight

FEBRUARY 19, 2020

Using variability in machine learning predictions as a proxy for risk can help studio executives and producers decide whether or not to green light a film project Photo by Kyle Smith on Unsplash Originally posted on Toward Data Science. Are you interested in working on high-impact projects and transitioning to a career in data?

Risk 67

Risk ROI Modeling Metrics 67

The Unofficial Google Data Science Blog

JANUARY 14, 2016

by AMIR NAJMI Running live experiments on large-scale online services (LSOS) is an important aspect of data science. We must therefore maintain statistical rigor in quantifying experimental uncertainty. In this post we explore how and why we can be “ data-rich but information-poor ”.

Experimentation 41

Experimentation Statistics Metrics Measurement 41

CIO Business Intelligence

JANUARY 4, 2023

In the HPC community, we recognize a need for tools to support machine learning operations and data science management; these tools must be able to scale and integrate with HPC software, compute and storage environments.

Cost-Benefit 57

Cost-Benefit Machine Learning Modeling Uncertainty 57

AWS Big Data

AUGUST 7, 2023

Many businesses use different software tools to analyze historical data and past patterns to forecast future demand and trends to make more accurate financial, marketing, and operational decisions. Forecasting acts as a planning tool to help enterprises prepare for the uncertainty that can occur in the future.

Forecasting 86

Forecasting Data Warehouse Machine Learning Modeling 86

The Unofficial Google Data Science Blog

APRIL 16, 2019

Such decisions involve an actual hypothesis test on specific metrics (e.g. Often, an established product will have an overall evaluation criterion (OEC) that incorporates trade-offs among important metrics and between short- and long-term success. The metrics to measure the impact of the change might not yet be established.

Experimentation 68

Experimentation Sales Metrics Measurement 68

The Unofficial Google Data Science Blog

APRIL 19, 2021

Calibration and other considerations Calibration is a desirable property, but it is not the only important metric. bar{pi} (1 - bar{pi})$: This is the irreducible loss due to uncertainty. Other important losses we consider are accuracy (the proportion of correct classifications) and discrimination based metrics like AUC.

Modeling 122

Modeling IT Metrics Testing 122

Domino Data Lab

JUNE 23, 2019

Companies with successful ML projects are often companies that already have an experimental culture in place as well as analytics that enable them to learn from data. Ensure that product managers work on projects that matter to the business and/or are aligned to strategic company metrics. That’s another pattern.

Management 110

Management Machine Learning Experimentation Metrics 110

Peter James Thomas

AUGUST 12, 2018

Again see Using BI to drive improvements in data quality for further details. Pertinence and fidelity of metrics developed from Data. Here we get past issues with data itself (or how it is handled and moved around) and instead consider how it is used. There are often compromises to be made in defining metrics.

Metrics 49

Metrics Statistics Data Quality Measurement 49

The Unofficial Google Data Science Blog

FEBRUARY 29, 2016

The result is that experimenters can’t afford to be sloppy about quantifying uncertainty. These typically result in smaller estimation uncertainty and tighter interval estimates. Variance reduction through conditioning Suppose, as an LSOS experimenter, you find that your key metric varies a lot by country and time of day.

Experimentation 40

Experimentation Metrics Statistics Measurement 40

The Unofficial Google Data Science Blog

OCTOBER 21, 2015

Our post describes how we arrived at recent changes to design principles for the Google search page, and thus highlights aspects of a data scientist’s role which involve practicing the scientific method. There has been debate as to whether the term “data science” is necessary. Some don’t see the point.

Slice and Dice 45

Slice and Dice Experimentation Data-driven Data Science 45

The Unofficial Google Data Science Blog

MAY 31, 2016

This means it is possible to specify exactly in which geos an ad campaign will be served – and to observe the ad spend and the response metric at the geo level. In other words, iROAS is the slope of a curve of the response metric plotted against the underlying advertising spend. They are non-overlapping geo-targetable regions.

Advertising 42

Advertising Testing Sales Statistics 42