Remove Finance Remove Knowledge Discovery Remove Metrics Remove Uncertainty
article thumbnail

Variance and significance in large-scale online services

The Unofficial Google Data Science Blog

Unlike experimentation in some other areas, LSOS experiments present a surprising challenge to statisticians — even though we operate in the realm of “big data”, the statistical uncertainty in our experiments can be substantial. We must therefore maintain statistical rigor in quantifying experimental uncertainty.

article thumbnail

LSOS experiments: how I learned to stop worrying and love the variability

The Unofficial Google Data Science Blog

The result is that experimenters can’t afford to be sloppy about quantifying uncertainty. These typically result in smaller estimation uncertainty and tighter interval estimates. Variance reduction through conditioning Suppose, as an LSOS experimenter, you find that your key metric varies a lot by country and time of day.