Remove Data Science Remove Knowledge Discovery Remove Metrics Remove Uncertainty
article thumbnail

Variance and significance in large-scale online services

The Unofficial Google Data Science Blog

by AMIR NAJMI Running live experiments on large-scale online services (LSOS) is an important aspect of data science. We must therefore maintain statistical rigor in quantifying experimental uncertainty. In this post we explore how and why we can be “ data-rich but information-poor ”.

article thumbnail

LSOS experiments: how I learned to stop worrying and love the variability

The Unofficial Google Data Science Blog

The result is that experimenters can’t afford to be sloppy about quantifying uncertainty. These typically result in smaller estimation uncertainty and tighter interval estimates. Variance reduction through conditioning Suppose, as an LSOS experimenter, you find that your key metric varies a lot by country and time of day.