The Unofficial Google Data Science Blog

JANUARY 14, 2016

by AMIR NAJMI Running live experiments on large-scale online services (LSOS) is an important aspect of data science. We must therefore maintain statistical rigor in quantifying experimental uncertainty. In this post we explore how and why we can be “ data-rich but information-poor ”. And an LSOS is awash in data, right?

Experimentation 41

Experimentation Statistics Metrics Measurement 41

The Unofficial Google Data Science Blog

JULY 22, 2020

Instead, we focus on the case where an experimenter has decided to run a full traffic ramp-up experiment and wants to use the data from all of the epochs in the analysis. When there are changing assignment weights and time-based confounders, this complication must be considered either in the analysis or the experimental design.

Experimentation 95

Experimentation Statistics Testing Strategy 95

The Unofficial Google Data Science Blog

FEBRUARY 29, 2016

Despite a very large number of experimental units, the experiments conducted by LSOS cannot presume statistical significance of all effects they deem practically significant. The result is that experimenters can’t afford to be sloppy about quantifying uncertainty. At Google, we tend to refer to them as slices.

Experimentation 40

Experimentation Metrics Statistics Measurement 40