The Unofficial Google Data Science Blog

JANUARY 14, 2016

Unlike experimentation in some other areas, LSOS experiments present a surprising challenge to statisticians — even though we operate in the realm of “big data”, the statistical uncertainty in our experiments can be substantial. We must therefore maintain statistical rigor in quantifying experimental uncertainty.

Experimentation 41

Experimentation Statistics Metrics Measurement 41

The Unofficial Google Data Science Blog

JULY 22, 2020

This post considers a common design for an OCE where a user may be randomly assigned an arm on their first visit during the experiment, with assignment weights referring to the proportion that are randomly assigned to each arm. There are two common reasons assignment weights may change during an OCE.

Experimentation 95

Experimentation Statistics Testing Strategy 95

The Unofficial Google Data Science Blog

FEBRUARY 29, 2016

Despite a very large number of experimental units, the experiments conducted by LSOS cannot presume statistical significance of all effects they deem practically significant. The result is that experimenters can’t afford to be sloppy about quantifying uncertainty. At Google, we tend to refer to them as slices.

Experimentation 40

Experimentation Metrics Statistics Measurement 40