2007, Knowledge Discovery and Measurement

Experiment design and modeling for long-term studies in ads

The Unofficial Google Data Science Blog

OCTOBER 7, 2015

by HENNING HOHNHOLD, DEIRDRE O'BRIEN, and DIANE TANG In this post we discuss the challenges in measuring and modeling the long-term effect of ads on user behavior. Nevertheless, A/B testing has challenges and blind spots, such as: the difficulty of identifying suitable metrics that give "works well" a measurable meaning.

Modeling

Modeling Experimentation Knowledge Discovery Testing

Changing assignment weights with time-based confounders

The Unofficial Google Data Science Blog

JULY 22, 2020

For this reason we don’t report uncertainty measures or statistical significance in the results of the simulation. Ramp-up solution: measure epoch and condition on its effect If one wants to do full traffic ramp-up and use data from all epochs, they must use an adjusted estimator to get an unbiased estimate of the average reward in each arm.

Experimentation

Experimentation Statistics Testing Strategy

Data Leaders Brief

Experiment design and modeling for long-term studies in ads

Changing assignment weights with time-based confounders

Webinars

Stay Connected