article thumbnail

Towards optimal experimentation in online systems

The Unofficial Google Data Science Blog

If the relationship of $X$ to $Y$ can be approximated as quadratic (or any polynomial), the objective and constraints as linear in $Y$, then there is a way to express the optimization as a quadratically constrained quadratic program (QCQP). However, joint optimization is possible by increasing both $x_1$ and $x_2$ at the same time.

article thumbnail

Experiment design and modeling for long-term studies in ads

The Unofficial Google Data Science Blog

This is essentially the same as finding a truly useful objective to optimize. accounting for effects "orthogonal" to the randomization used in experimentation. The first thing you’ll want to do is to run your test for a long time with fixed experimental units, in our case cookies.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

To Balance or Not to Balance?

The Unofficial Google Data Science Blog

In an ideal world, experimentation through randomization of the treatment assignment allows the identification and consistent estimation of causal effects. It should be noted that inverse probability weighting is not generally optimal (i.e., 2007): Propose a finite collection $mathcal L={hat e_k:k=1,ldots,K}$ of estimation algorithms.

article thumbnail

Knowledge

Occam's Razor

The Awesome Power of Visualization 2 -> Death and Taxes 2007. Build A Great Web Experimentation & Testing Program. Experimentation and Testing: A Primer. Search Engine Optimization (SEO) Metrics & Analytics. 2007 Predictions: Web Analytics. Web Analytics Demystified. Six Data Visualizations That Rock!

KPI 124
article thumbnail

The Lean Analytics Cycle: Metrics > Hypothesis > Experiment > Act

Occam's Razor

Sometimes, we escape the clutches of this sub optimal existence and do pick good metrics or engage in simple A/B testing. You're choosing only one metric because you want to optimize it. Circle of Friends was a social community built atop Facebook that launched in 2007. But it is not routine. So, how do we fix this problem?

Metrics 156
article thumbnail

Measuring Incrementality: Controlled Experiments to the Rescue!

Occam's Razor

We have to do Search Engine Optimization. Bonus: Here's one of my favorite articles… all the way from 2007 but chock full of pithy valuable lessons for all of us regardless of our field: 41 Timeless Ways to Screw Up Direct Marketing. Having read this post what might be the biggest downside to experimentation?

article thumbnail

Changing assignment weights with time-based confounders

The Unofficial Google Data Science Blog

Instead, we focus on the case where an experimenter has decided to run a full traffic ramp-up experiment and wants to use the data from all of the epochs in the analysis. When there are changing assignment weights and time-based confounders, this complication must be considered either in the analysis or the experimental design.