article thumbnail

Towards optimal experimentation in online systems

The Unofficial Google Data Science Blog

the weight given to Likes in our video recommendation algorithm) while $Y$ is a vector of outcome measures such as different metrics of user experience (e.g., Crucially, it takes into account the uncertainty inherent in our experiments. Figure 2: Spreading measurements out makes estimates of model (slope of line) more accurate.

article thumbnail

Uncertainties: Statistical, Representational, Interventional

The Unofficial Google Data Science Blog

by AMIR NAJMI & MUKUND SUNDARARAJAN Data science is about decision making under uncertainty. Some of that uncertainty is the result of statistical inference, i.e., using a finite sample of observations for estimation. But there are other kinds of uncertainty, at least as important, that are not statistical in nature.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Business Strategies for Deploying Disruptive Tech: Generative AI and ChatGPT

Rocket-Powered Data Science

Those F’s are: Fragility, Friction, and FUD (Fear, Uncertainty, Doubt). encouraging and rewarding) a culture of experimentation across the organization. Encourage and reward a Culture of Experimentation that learns from failure, “ Test, or get fired! Test early and often. Expect continuous improvement.

Strategy 289
article thumbnail

What you need to know about product management for AI

O'Reilly on Data

Machine learning adds uncertainty. Underneath this uncertainty lies further uncertainty in the development process itself. There are strategies for dealing with all of this uncertainty–starting with the proverb from the early days of Agile: “ do the simplest thing that could possibly work.”

article thumbnail

Changing assignment weights with time-based confounders

The Unofficial Google Data Science Blog

Instead, we focus on the case where an experimenter has decided to run a full traffic ramp-up experiment and wants to use the data from all of the epochs in the analysis. When there are changing assignment weights and time-based confounders, this complication must be considered either in the analysis or the experimental design.

article thumbnail

How to create a culture of innovation

CIO Business Intelligence

Prioritize time for experimentation. It requires bold bets and a willingness to persevere despite setbacks, criticism, and uncertainty,’’ wrote McKinsey senior partners Laura Furstenthal and Erik Roth in a recent blog post. “By Here, they and others share seven ways to create and nurture a culture of innovation.

article thumbnail

Misadventures in experiments for growth

The Unofficial Google Data Science Blog

by MICHAEL FORTE Large-scale live experimentation is a big part of online product development. This means a small and growing product has to use experimentation differently and very carefully. This blog post is about experimentation in this regime. But these are not usually amenable to A/B experimentation.