The Unofficial Google Data Science Blog

OCTOBER 7, 2015

accounting for effects "orthogonal" to the randomization used in experimentation. For example in ads, experiments using cookies (users) as experimental units are not suited to capture the impact of a treatment on advertisers or publishers nor their reaction to it. To see this, imagine you want to study long-term effects in an A/B test.

Modeling 61

Modeling Experimentation Knowledge Discovery Testing 61

The Unofficial Google Data Science Blog

JANUARY 14, 2016

by AMIR NAJMI Running live experiments on large-scale online services (LSOS) is an important aspect of data science. We must therefore maintain statistical rigor in quantifying experimental uncertainty. In this post we explore how and why we can be “ data-rich but information-poor ”. And an LSOS is awash in data, right?

Experimentation 41

Experimentation Statistics Metrics Measurement 41

The Unofficial Google Data Science Blog

JULY 22, 2020

Instead, we focus on the case where an experimenter has decided to run a full traffic ramp-up experiment and wants to use the data from all of the epochs in the analysis. Just as in ramp-up, making inferences while ignoring the complexity of time-based confounders that are present can lead to biased estimates.

Experimentation 95

Experimentation Statistics Testing Strategy 95