The Unofficial Google Data Science Blog

APRIL 23, 2024

the weight given to Likes in our video recommendation algorithm) while $Y$ is a vector of outcome measures such as different metrics of user experience (e.g., Crucially, it takes into account the uncertainty inherent in our experiments. Figure 2: Spreading measurements out makes estimates of model (slope of line) more accurate.

Experimentation 52

Experimentation Optimization Uncertainty Metrics 52

The Unofficial Google Data Science Blog

DECEMBER 14, 2021

by AMIR NAJMI & MUKUND SUNDARARAJAN Data science is about decision making under uncertainty. Some of that uncertainty is the result of statistical inference, i.e., using a finite sample of observations for estimation. But there are other kinds of uncertainty, at least as important, that are not statistical in nature.

Uncertainty 56

Uncertainty Statistics Measurement Cost-Benefit 56

Rocket-Powered Data Science

FEBRUARY 15, 2023

Those F’s are: Fragility, Friction, and FUD (Fear, Uncertainty, Doubt). encouraging and rewarding) a culture of experimentation across the organization. A business-disruptive ChatGPT implementation definitely fits into this category: focus first on the MVP or MLP. Test early and often. Expect continuous improvement.

Strategy 289

Strategy Experimentation Uncertainty Machine Learning 289

O'Reilly on Data

OCTOBER 13, 2020

This distinction assumes a slightly different definition of debugging than is often used in software development. Emmanuel Ameisen once again provides a useful and concise definition of model monitoring as a way to “track the health of a system. While useful, these constructs are not beyond criticism.

Management 244

Management Metrics Machine Learning Modeling 244

The Unofficial Google Data Science Blog

JANUARY 14, 2016

Unlike experimentation in some other areas, LSOS experiments present a surprising challenge to statisticians — even though we operate in the realm of “big data”, the statistical uncertainty in our experiments can be substantial. We must therefore maintain statistical rigor in quantifying experimental uncertainty.

Experimentation 41

Experimentation Statistics Metrics Measurement 41

Occam's Razor

APRIL 8, 2013

First, you figure out what you want to improve; then you create an experiment; then you run the experiment; then you measure the results and decide what to do. For each of them, write down the KPI you're measuring, and what that KPI should be for you to consider your efforts a success. Measure and decide what to do.

Metrics 156

Metrics KPI Analytics Key Performance Indicator 156

Domino Data Lab

JUNE 23, 2019

Skomoroch proposes that managing ML projects are challenging for organizations because shipping ML projects requires an experimental culture that fundamentally changes how many companies approach building and shipping software. These measurement-obsessed companies have an advantage when it comes to AI.

Management 110

Management Machine Learning Experimentation Metrics 110