Remove Finance Remove Knowledge Discovery Remove Measurement Remove Testing
article thumbnail

Variance and significance in large-scale online services

The Unofficial Google Data Science Blog

Well, it turns out that depending on what it cares to measure, an LSOS might not have enough data. The practical consequence of this is that we can’t afford to be sloppy about measuring statistical significance and confidence intervals. For this purpose, let’s assume we use a t-test for difference between group means.

article thumbnail

Changing assignment weights with time-based confounders

The Unofficial Google Data Science Blog

Another reason to use ramp-up is to test if a website's infrastructure can handle deploying a new arm to all of its users. The website wants to make sure they have the infrastructure to handle the feature while testing if engagement increases enough to justify the infrastructure. Here, day-of-week is a time-based confounder.