Remove 2007 Remove Interactive Remove Metrics Remove Uncertainty
article thumbnail

The Lean Analytics Cycle: Metrics > Hypothesis > Experiment > Act

Occam's Razor

To win in business you need to follow this process: Metrics > Hypothesis > Experiment > Act. We are far too enamored with data collection and reporting the standard metrics we love because others love them because someone else said they were nice so many years ago. That metric is tied to a KPI.

Metrics 156
article thumbnail

Towards optimal experimentation in online systems

The Unofficial Google Data Science Blog

the weight given to Likes in our video recommendation algorithm) while $Y$ is a vector of outcome measures such as different metrics of user experience (e.g., Crucially, it takes into account the uncertainty inherent in our experiments. Here, $X$ is a vector of tuning parameters that control the system's operating characteristics (e.g.

article thumbnail

Why model calibration matters and how to achieve it

The Unofficial Google Data Science Blog

For these ML systems, calibration simplifies interaction. Calibration and other considerations Calibration is a desirable property, but it is not the only important metric. bar{pi} (1 - bar{pi})$: This is the irreducible loss due to uncertainty. And users may start receiving a lot more spam! This shows why $Pr(mathrm{Spam}) = 0.1$

Modeling 122