Occam's Razor

MAY 22, 2006

This post is a primer on the delightful world of testing and experimentation (A/B, Multivariate, and a new term from me: Experience Testing). Experimentation and testing help us figure out we are wrong, quickly and repeatedly and if you think about it that is a great thing for our customers, and for our employers. Counter claims?

Experimentation 103

Experimentation Testing Optimization Measurement 103

The Unofficial Google Data Science Blog

OCTOBER 7, 2015

by HENNING HOHNHOLD, DEIRDRE O'BRIEN, and DIANE TANG In this post we discuss the challenges in measuring and modeling the long-term effect of ads on user behavior. Nevertheless, A/B testing has challenges and blind spots, such as: the difficulty of identifying suitable metrics that give "works well" a measurable meaning.

Modeling 61

Modeling Experimentation Knowledge Discovery Testing 61

The Unofficial Google Data Science Blog

JANUARY 14, 2016

Unlike experimentation in some other areas, LSOS experiments present a surprising challenge to statisticians — even though we operate in the realm of “big data”, the statistical uncertainty in our experiments can be substantial. We must therefore maintain statistical rigor in quantifying experimental uncertainty.

Experimentation 41

Experimentation Statistics Metrics Measurement 41

DataRobot Blog

JULY 28, 2022

In addition to the accuracy of the models we built, we had to consider business metrics, cost, interpretability, and suitability for ongoing operations. For example, data measured by sensors can contain all kinds of noise due to sensor malfunctions, environmental changes, etc., which can lead to large prediction errors.

Data Science 72

Data Science Machine Learning Statistics Modeling 72

Occam's Razor

JULY 24, 2018

work (collection, processing, reporting, analysis), processes, org structure, governance models, last-mile gaps , metrics ladders of awesomeness , and… so… much… more. Remember, tools, work, processes, org structure, governance models, last-mile gaps, metrics ladders of awesomeness, and… so… much… more. The Implications of Complexity.

Analytics 93

Analytics Metrics Strategy Measurement 93

The Unofficial Google Data Science Blog

JANUARY 16, 2018

Experimentation on networks A/B testing is a standard method of measuring the effect of changes by randomizing samples into different treatment groups. With A/B testing, we can validate various hypotheses and measure the impact of our product changes, allowing us to make better decisions. This could create confusion.

Testing 58

Testing Experimentation Measurement Modeling 58

Sisense

JANUARY 21, 2020

One of the simplest ways to start exploring your data is to aggregate the metrics you are interested in by their relevant dimensions. They measured both the blood pressure of the participants and if they had a heart attack or not. Our BI Best Practices demystify the analytics world and empower you with actionable how-to guidance.

Testing 104

Testing Risk Data-driven Statistics 104

Cloudera

FEBRUARY 16, 2022

blueberry spacing) is a measure of the model’s interpretability. Support for multiple sessions within a project allows data scientists, engineers and operations teams to work independently alongside each other on experimentation, pipeline development, deployment and monitoring activities in parallel. Model Visibility.

Machine Learning 81

Machine Learning Modeling Metadata Recreation/Entertainment 81

Occam's Razor

NOVEMBER 13, 2009

I now have several full time jobs, plus this blog, plus speaking around the world, plus a family, plus… so much more. " This blog post is in three parts: The pitch. Chapter 3 The Awesome World of Clickstream Analysis: Metrics. Chapter 5 The Key to Glory: Measuring Success. Request for help. A good thing.

Analytics 71

Analytics Metrics Measurement Testing 71

O'Reilly on Data

MAY 14, 2020

AI PMs should enter feature development and experimentation phases only after deciding what problem they want to solve as precisely as possible, and placing the problem into one of these categories. Experimentation: It’s just not possible to create a product by building, evaluating, and deploying a single model.

Management 363

Management Experimentation B2B Machine Learning 363

datapine

FEBRUARY 4, 2020

Management thinker Peter Drucker once stated, “if you can’t measure it, you can’t improve it” – and he couldn’t be more right. Structure your metrics. As with any report you might need to create, structuring and implementing metrics that will tell an interesting and educational data-story is crucial in our digital age.

Reporting 280

Reporting Marketing Advertising Metrics 280

O'Reilly on Data

OCTOBER 13, 2020

Ideally, AI PMs would steer development teams to incorporate I/O validation into the initial build of the production system, along with the instrumentation needed to monitor model accuracy and other technical performance metrics. But in practice, it is common for model I/O validation steps to be added later, when scaling an AI product.

Management 244

Management Metrics Machine Learning Modeling 244

Occam's Razor

FEBRUARY 26, 2008

1: Implement a Experimentation & Testing Program. # 1: Implement a Experimentation & Testing Program. Experimentation and Testing: A Primer. Build A Great Web Experimentation & Testing Program. # Be it for in vogue metrics like Conversion Rates or for metrics that should be in vogue like Abandonment Rates.

Management 68

Management Experimentation Testing Strategy 68

Occam's Razor

JUNE 6, 2007

It has been such an amazing journey to write the book, and for it to come up almost exactly a year after I started this blog. Bonus: Interactive CD: Contains six podcasts, one video, two web analytics metrics definitions documents and five insightful powerpoint presentations. There I said it. Damini, Chirag and now the book! :).

Analytics 60

Analytics Metrics Marketing Dashboards 60

The Unofficial Google Data Science Blog

DECEMBER 14, 2021

This blog post introduces the notions of representational uncertainty and interventional uncertainty to paint a fuller picture of what the practicing data scientist is up against. Representational uncertainty : the gap between the desired meaning of some measure and its actual meaning. This blog post does not discuss objectives.

Uncertainty 56

Uncertainty Statistics Measurement Cost-Benefit 56

The Unofficial Google Data Science Blog

APRIL 16, 2019

by MICHAEL FORTE Large-scale live experimentation is a big part of online product development. In fact, this blog has published posts on this very topic. This means a small and growing product has to use experimentation differently and very carefully. This blog post is about experimentation in this regime.

Experimentation 68

Experimentation Sales Metrics Measurement 68

DataKitchen

APRIL 13, 2021

Read the complete blog below for a more detailed description of the vendors and their capabilities. DataOps requires that teams measure their analytic processes in order to see how they are improving over time. A complete DataOps program will have a unified, system-wide view of process metrics using a common data store.

Testing 300

Testing Machine Learning Consulting Data Quality 300

The Unofficial Google Data Science Blog

JUNE 30, 2016

In an ideal world, experimentation through randomization of the treatment assignment allows the identification and consistent estimation of causal effects. We use performance metrics such as bias and mean squared error for the estimation of $delta$, our causal estimand of interest, defined as the average effect of treatment on the treated.

Statistics 44

Statistics Optimization Modeling Experimentation 44

The Unofficial Google Data Science Blog

JULY 31, 2016

The analysis can be straightforward, especially when it's safe to assume that individual observations of an outcome measure are independent. This blog post explores how this problem arises in applications at Google, and how we can 'mind our units' through analysis and logging. When analyzing the outcome measure (e.g.,

Experimentation 42

Experimentation Testing Measurement Metrics 42

The Unofficial Google Data Science Blog

FEBRUARY 29, 2016

Despite a very large number of experimental units, the experiments conducted by LSOS cannot presume statistical significance of all effects they deem practically significant. The result is that experimenters can’t afford to be sloppy about quantifying uncertainty. In statistics, such segments are often called “blocks” or “strata”.

Experimentation 40

Experimentation Metrics Statistics Measurement 40

The Unofficial Google Data Science Blog

MARCH 22, 2018

But what if users don't immediately uptake the new experimental version? This blog post provides details for how we can make inferences without waiting for complete uptake. This blog post provides details for how we can make inferences without waiting for complete uptake. What if their uptake rate is not uniform?

Experimentation 62

Experimentation Measurement Modeling Statistics 62

Occam's Razor

JULY 26, 2010

If today you are a content site that is only focused on measuring content consumed try to go deeper to understanding CPA of the ads or Visitor Loyalty. PALM: People Against Lonely Metrics]. So why not your metrics? This is the problem with lonely metrics. Never ever never never never ever present any metric all by itself.

Metrics 96

Metrics Measurement Dashboards Reporting 96

Occam's Razor

AUGUST 13, 2012

Yehoshua Coren: Best ways to measure user behavior in a multi-touch, multi-device digital world. Yehoshua I've covered this topic in detail in this blog post: Multi-Channel Attribution: Definitions, Models and a Reality Check. What's possible to measure. What's not possible to measure. Let's do this!

Modeling 124

Modeling Metrics Data Quality Data-driven 124

Occam's Razor

MARCH 23, 2010

Too many new things are happening too fast and those of us charged with measuring it have to change the wheels while the bicycle is moving at 30 miles per hour (and this bicycle will become a car before we know it – all while it keeps moving, ever faster). It has simply not had a break to catch a breath and mature. Likely not.

Analytics 118

Analytics Consulting Metrics Experimentation 118

Occam's Razor

SEPTEMBER 30, 2013

We'll start with digital at the highest strategic level, which leads us into content marketing, from there it is a quick hop over to the challenge of metrics and silos, followed by a recommendation to optimize for the global maxima, and we end with the last two visuals that cover social investment and social content strategy.

Marketing Analytics 154

Marketing Analytics Marketing Visualization Analytics 154

The Unofficial Google Data Science Blog

MAY 31, 2016

It is important that we can measure the effect of these offline conversions as well. Panel studies make it possible to measure user behavior along with the exposure to ads and other online elements. Let's take a look at larger groups of individuals whose aggregate behavior we can measure. days or weeks).

Advertising 42

Advertising Testing Sales Statistics 42

Occam's Razor

FEBRUARY 7, 2011

After 416,350 words in posts and 845,128 words in comments on this blog, thus far, there is always more to explore, illuminate and share. It is a great way to stay connected to what's most important to you (and keep the blog and its content relevant!). But each keyword gets "credit" for other metrics.

Analytics 94

Analytics Measurement Metrics Predictive Analytics 94

Domino Data Lab

FEBRUARY 4, 2019

Spoiler alert: a research field called curiosity-driven learning is emerging at the nexis of experimental cognitive psychology and industry use cases for machine learning, particularly in gaming AI. The ability to measure results (risk-reducing evidence). Ensure a culture that supports a steady process of learning and experimentation.

Data Science 66

Data Science Experimentation Data-driven Machine Learning 66

Occam's Razor

MAY 18, 2010

way of identify actionable insights is on display in pretty much every single blog post I write. One of my earliest blog posts extolled the glorious virtues of segmentation: Excellent Analytics Tip#2: Segment Absolutely Everything. Key elements of the Web Analytics Measurement Framework.]. The Problem. All visits. Total revenue.

Analytics 123

Analytics Key Performance Indicator Metrics Measurement 123

Occam's Razor

DECEMBER 12, 2016

Digital Marketing & Measurement Model. My solution to this, incredibly real and frustrating problem, is to insist on seeing the signed in blood version of the company's Digital Marketing & Measurement Model. What one critical metric will help you clearly measure performance for each strategy above?

Analytics 141

Analytics Data-driven Strategy Consulting 141

Occam's Razor

JANUARY 9, 2012

Most companies are astonishingly blasé about data and possibilities of measurement. " Sad, unimaginative measurements of their sad, unimaginative campaigns. You've probably seen these rules sneak into my blog posts. I've talked about the long tail on this blog, especially in context for Search.

Marketing 140

Marketing B2B Measurement Risk 140

The Unofficial Google Data Science Blog

OCTOBER 21, 2015

We’re not going to engage in this debate but in this blog post we do focus on science. It is important to make clear distinctions among each of these, and to advance the state of knowledge through concerted observation, modeling and experimentation. This is very much the work of a scientist.

Slice and Dice 45

Slice and Dice Experimentation Data-driven Data Science 45

Domino Data Lab

MAY 8, 2019

Finale Doshi-Velez, Been Kim (2017-02-28) ; see also the Domino blog article about TCAV. Visualizations are vital in data science work, with the caveat that the information that they convey may be 4-5 layers of abstraction away from the actual business process being measured. measure the subjects’ ability to trust the models’ results.

Machine Learning 105

Machine Learning Data Science Modeling Visualization 105

Cloudera

JUNE 6, 2019

At the other end of the spectrum, the admin may instantiate a number of low-priority dev clusters – these clusters may often run at capacity, not require performance guarantees, but also provide more agility and flexibility for experimentation. Designing a proper network for VPCs extends beyond the scope of this blog post.

Metadata 85

Metadata Data Lake Optimization Strategy 85

Occam's Razor

OCTOBER 19, 2010

I was reflecting on that recently and thought it was incredible that in all my years of writing this blog I have never written a blog post, not one single one (!!), My goal is to give you a list of tools that I use in my everyday life as a practitioner (you'll see many of them implemented on this blog). Disclosure].

Analytics 135

Analytics Testing Measurement Optimization 135

Occam's Razor

OCTOBER 23, 2006

Start with measuring these Outcomes metrics (revenue, leads, profit margins, improved product mix, number of new customers etc). Bonus: Here is a blog post that outlines in detail the difference between reporting and analysis, and shares ideas you can use every day: Rebel! 6 Reporting is not Analysis. Your Choice?

Data-driven 117

Data-driven Metrics KPI Reporting 117

Occam's Razor

JULY 1, 2008

3 ] Provide you with a bushel of specific multichannel measurement ideas to help quantify the offline impact of your online presence. It's a tall order, but after two years of blogging why stop now. : ). Why should you care about measuring multichannel impact? But also be pragmatic, about how much, how accurate and when.

Analytics 116

Analytics Measurement Marketing Sales 116