O'Reilly on Data

OCTOBER 13, 2020

Ideally, AI PMs would steer development teams to incorporate I/O validation into the initial build of the production system, along with the instrumentation needed to monitor model accuracy and other technical performance metrics. But in practice, it is common for model I/O validation steps to be added later, when scaling an AI product.

Management 244

Management Metrics Machine Learning Modeling 244

Domino Data Lab

JUNE 2, 2019

Co-chair Paco Nathan provides highlights of Rev 2 , a data science leaders summit. We held Rev 2 May 23-24 in NYC, as the place where “data science leaders and their teams come to learn from each other.” If you lead a data science team/org, DM me and I’ll send you an invite to data-head.slack.com ”.

Data-driven 110

Data-driven Data Science Machine Learning Modeling 110

The Unofficial Google Data Science Blog

AUGUST 27, 2019

Of course, any mistakes by the reviewers would propagate to the accuracy of the metrics, and the metrics calculation should take into account human errors. If we could separate bad videos from good videos perfectly, we could simply calculate the metrics directly without sampling. The missing verdicts create two problems.

Metrics 98

Metrics Statistics Uncertainty Optimization 98

Peter James Thomas

AUGUST 12, 2018

This piece was prompted by both Olaf’s question and a recent article by my friend Neil Raden on his Silicon Angle blog, Performance management: Can you really manage what you measure? Data may be perfectly valid, but still not represent reality. It is hard to account for such tweaking in measurement systems.

Metrics 49

Metrics Statistics Data Quality Measurement 49

Domino Data Lab

JUNE 23, 2019

Companies with successful ML projects are often companies that already have an experimental culture in place as well as analytics that enable them to learn from data. Ensure that product managers work on projects that matter to the business and/or are aligned to strategic company metrics. That’s another pattern.

Management 110

Management Machine Learning Experimentation Metrics 110

Insight

FEBRUARY 19, 2020

Using variability in machine learning predictions as a proxy for risk can help studio executives and producers decide whether or not to green light a film project Photo by Kyle Smith on Unsplash Originally posted on Toward Data Science. Hollywood is a $10 billion-a-year industry, and movies range from huge hits to box office bombs.

Risk 67

Risk ROI Modeling Metrics 67

The Unofficial Google Data Science Blog

APRIL 16, 2019

Such decisions involve an actual hypothesis test on specific metrics (e.g. Often, an established product will have an overall evaluation criterion (OEC) that incorporates trade-offs among important metrics and between short- and long-term success. The metrics to measure the impact of the change might not yet be established.

Experimentation 68

Experimentation Sales Metrics Measurement 68

The Unofficial Google Data Science Blog

MAY 31, 2016

It is important that we can measure the effect of these offline conversions as well. Panel studies make it possible to measure user behavior along with the exposure to ads and other online elements. Wouldn't it be great if we didn't require individual data to estimate an aggregate effect? days or weeks).

Advertising 42

Advertising Testing Sales Statistics 42

The Unofficial Google Data Science Blog

APRIL 19, 2021

The numerical value of the signal became decoupled from the event it was measuring even as the ordinal value remained unchanged. Calibration and other considerations Calibration is a desirable property, but it is not the only important metric. bar{pi} (1 - bar{pi})$: This is the irreducible loss due to uncertainty.

Modeling 122

Modeling IT Metrics Testing 122

The Unofficial Google Data Science Blog

FEBRUARY 29, 2016

The result is that experimenters can’t afford to be sloppy about quantifying uncertainty. These typically result in smaller estimation uncertainty and tighter interval estimates. Variance reduction through conditioning Suppose, as an LSOS experimenter, you find that your key metric varies a lot by country and time of day.

Experimentation 40

Experimentation Metrics Statistics Measurement 40

The Unofficial Google Data Science Blog

OCTOBER 21, 2015

Our post describes how we arrived at recent changes to design principles for the Google search page, and thus highlights aspects of a data scientist’s role which involve practicing the scientific method. There has been debate as to whether the term “data science” is necessary. Some don’t see the point.

Slice and Dice 45

Slice and Dice Experimentation Data-driven Data Science 45