article thumbnail

Towards optimal experimentation in online systems

The Unofficial Google Data Science Blog

Crucially, it takes into account the uncertainty inherent in our experiments. To find optimal values of two parameters experimentally, the obvious strategy would be to experiment with and update them in separate, sequential stages. In this section we’ll discuss how we approach these two kinds of uncertainty with QCQP.

article thumbnail

Changing assignment weights with time-based confounders

The Unofficial Google Data Science Blog

Instead, we focus on the case where an experimenter has decided to run a full traffic ramp-up experiment and wants to use the data from all of the epochs in the analysis. When there are changing assignment weights and time-based confounders, this complication must be considered either in the analysis or the experimental design.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

The Lean Analytics Cycle: Metrics > Hypothesis > Experiment > Act

Occam's Razor

Circle of Friends was a social community built atop Facebook that launched in 2007. They might deal with uncertainty, but they're not random. Advanced Analytics Big Data Digital Analytics Web Analytics Web Insights Web Metrics actionable analytics business optimization experimentation and testing key performance indicators'

Metrics 156
article thumbnail

The trinity of errors in applying confidence intervals: An exploration using Statsmodels

O'Reilly on Data

Because of this trifecta of errors, we need dynamic models that quantify the uncertainty inherent in our financial estimates and predictions. Practitioners in all social sciences, especially financial economics, use confidence intervals to quantify the uncertainty in their estimates and predictions.

article thumbnail

Estimating causal effects using geo experiments

The Unofficial Google Data Science Blog

A geo experiment is an experiment where the experimental units are defined by geographic regions. The expected precision of our inferences can be computed by simulating possible experimental outcomes. Cambridge, 2007. Wouldn't it be great if we didn't require individual data to estimate an aggregate effect?