Occam's Razor

MAY 22, 2006

This post is a primer on the delightful world of testing and experimentation (A/B, Multivariate, and a new term from me: Experience Testing). Experimentation and testing help us figure out we are wrong, quickly and repeatedly and if you think about it that is a great thing for our customers, and for our employers. Counter claims?

Experimentation 103

Experimentation Testing Optimization Measurement 103

DataRobot Blog

OCTOBER 18, 2022

Model Observability – the ability to track key health and service metrics for models in production – remains a top priority for AI-enabled organizations. Learn more about the DataRobot AI Cloud and the ability to accelerate your experimentation and production timelines. DataRobot Booth at Big Data & AI Toronto 2022.

Big Data 96

Big Data Forecasting Experimentation Metrics 96

Cloudera

APRIL 9, 2021

This is part 4 in this blog series. This blog series follows the manufacturing and operations data lifecycle stages of an electric car manufacturer – typically experienced in large, data-driven manufacturing companies. The second blog dealt with creating and managing Data Enrichment pipelines. Here are the key stages: .

Machine Learning 98

Machine Learning Forecasting Manufacturing Predictive Analytics 98

Occam's Razor

MAY 15, 2006

There are many types of qualitative data at your disposal including brand buzz, customer satisfaction, net promoter indices, visitor engagement, stickiness, blog-pulse, etc. There is a lot of "buzz" around "buzzy" metrics such as brand value / brand impact, blog-pulse , to name a couple.

Metrics 100

Metrics Experimentation Testing Interactive 100

IBM Big Data Hub

MARCH 4, 2024

This culture encourages experimentation and expertise growth. Visit the IBM Hybrid Cloud Architecture Center The post How to become an AI+ enterprise appeared first on IBM Blog. For example, by using compliance control scanning of terraform templates to fail provisioning if controls are not met.

Enterprise 112

Enterprise ROI Risk Strategy 112

DataRobot Blog

FEBRUARY 28, 2023

They can enjoy a hosted experience with code snippets, versioning, and simple environment management for rapid AI experimentation. When the training process is complete, DataRobot will recommend the best-performing model for production based on the selected metric and provide an explanation. Learn more about DataRobot hosted notebooks.

Data Processing 72

Data Processing Experimentation Machine Learning Modeling 72

Occam's Razor

AUGUST 22, 2011

Here you'll find all my blog posts categorized into a structure that will hopefully make it easy for you to discover new content, find answers to your questions, or simply wallow in some excellent analytics narratives. " ~ Web Metrics: "What is a KPI? " + Standard Metrics Revisited Series. How to focus?"

KPI 124

KPI Metrics Measurement ROI 124

The Unofficial Google Data Science Blog

JANUARY 14, 2016

Unlike experimentation in some other areas, LSOS experiments present a surprising challenge to statisticians — even though we operate in the realm of “big data”, the statistical uncertainty in our experiments can be substantial. We must therefore maintain statistical rigor in quantifying experimental uncertainty.

Experimentation 41

Experimentation Statistics Metrics Measurement 41

DataRobot Blog

JANUARY 10, 2023

Data science teams of all sizes need a productive, collaborative method for rapid AI experimentation. This flexibility allows you to import your local code into the DataRobot platform and continue further experimentation using the combination of DataRobot Notebooks with: Deep integrations with DataRobot comprehensive APIs.

Experimentation 52

Experimentation Machine Learning Data Science Modeling 52

DataRobot Blog

JULY 28, 2022

In addition to the accuracy of the models we built, we had to consider business metrics, cost, interpretability, and suitability for ongoing operations. Initially, the customer tried modeling using statistical methods to create typical features, such as moving averages, but the model metrics (R-square) was only 0.5

Data Science 72

Data Science Machine Learning Statistics Modeling 72

The Unofficial Google Data Science Blog

OCTOBER 7, 2015

Nevertheless, A/B testing has challenges and blind spots, such as: the difficulty of identifying suitable metrics that give "works well" a measurable meaning. accounting for effects "orthogonal" to the randomization used in experimentation. accounting for effects "orthogonal" to the randomization used in experimentation.

Modeling 61

Modeling Experimentation Knowledge Discovery Testing 61

Occam's Razor

JULY 24, 2018

work (collection, processing, reporting, analysis), processes, org structure, governance models, last-mile gaps , metrics ladders of awesomeness , and… so… much… more. Remember, tools, work, processes, org structure, governance models, last-mile gaps, metrics ladders of awesomeness, and… so… much… more. The Implications of Complexity.

Analytics 93

Analytics Metrics Strategy Measurement 93

The Unofficial Google Data Science Blog

JANUARY 16, 2018

Experimentation on networks A/B testing is a standard method of measuring the effect of changes by randomizing samples into different treatment groups. However, the downside of using a larger unit of randomization is that we lose experimental power. Consider the case where experiment metrics are evaluated at the per-user level.

Testing 58

Testing Experimentation Measurement Modeling 58

Sisense

JANUARY 21, 2020

One of the simplest ways to start exploring your data is to aggregate the metrics you are interested in by their relevant dimensions. Another study showed using experimental studies that the paradox might occur, and that people are often poor at recognizing it. See “ How likely is Simpson’s paradox?

Testing 104

Testing Risk Data-driven Statistics 104

Cloudera

FEBRUARY 16, 2022

Support for multiple sessions within a project allows data scientists, engineers and operations teams to work independently alongside each other on experimentation, pipeline development, deployment and monitoring activities in parallel. The post Of Muffins and Machine Learning Models appeared first on Cloudera Blog.

Machine Learning 81

Machine Learning Modeling Metadata Recreation/Entertainment 81

Domino Data Lab

AUGUST 6, 2021

Develop: includes accessing and preparing data and algorithms, researching and development of models and experimentation. Monitor: includes monitoring the performance of the model, tracking metrics, as well as driving adoption of the model by those it was intended to serve.

Data Science 75

Data Science Testing Modeling Publishing 75

O'Reilly on Data

MAY 14, 2020

AI PMs should enter feature development and experimentation phases only after deciding what problem they want to solve as precisely as possible, and placing the problem into one of these categories. Experimentation: It’s just not possible to create a product by building, evaluating, and deploying a single model.

Management 363

Management Experimentation B2B Machine Learning 363

Occam's Razor

NOVEMBER 13, 2009

I now have several full time jobs, plus this blog, plus speaking around the world, plus a family, plus… so much more. " This blog post is in three parts: The pitch. Chapter 3 The Awesome World of Clickstream Analysis: Metrics. Chapter 7 Failing Faster: Unleashing the Power of Testing and Experimentation.

Analytics 71

Analytics Metrics Measurement Testing 71

datapine

FEBRUARY 4, 2020

As we have already talked about in our previous blog post on sales reports for daily, weekly or monthly reporting, you need to figure out a couple of things when launching and executing a marketing campaign: are your efforts paying off? Structure your metrics. That way you can choose the best possible metrics for your case.

Reporting 280

Reporting Marketing Advertising Metrics 280

datapine

MAY 20, 2020

To inspire your customer relationship management report for managing your metrics, explore our cutting-edge selection of KPI examples. When we say “optimal design,” we don’t mean cramming piles of information into one space or being overly experimental with colors. Work through your narrative.

Dashboards 265

Dashboards Reporting KPI Visualization 265

Occam's Razor

FEBRUARY 26, 2008

1: Implement a Experimentation & Testing Program. # 1: Implement a Experimentation & Testing Program. Experimentation and Testing: A Primer. Build A Great Web Experimentation & Testing Program. # Be it for in vogue metrics like Conversion Rates or for metrics that should be in vogue like Abandonment Rates.

Management 68

Management Experimentation Testing Strategy 68

Occam's Razor

JUNE 6, 2007

It has been such an amazing journey to write the book, and for it to come up almost exactly a year after I started this blog. Bonus: Interactive CD: Contains six podcasts, one video, two web analytics metrics definitions documents and five insightful powerpoint presentations. There I said it. Damini, Chirag and now the book! :).

Analytics 60

Analytics Metrics Marketing Dashboards 60

DataRobot Blog

MARCH 16, 2023

Collaborative Experimentation Experience – the new experience, called the Workbench, comes packed with new capabilities such as new integrated data prep for modeling and notebooks providing a full code-first experience. New Snowflake integrations and the SAP joint solution have tightened the data to experimentation to deployment loop.

Experimentation 98

Experimentation Data-driven Modeling Metadata 98

O'Reilly on Data

OCTOBER 13, 2020

Ideally, AI PMs would steer development teams to incorporate I/O validation into the initial build of the production system, along with the instrumentation needed to monitor model accuracy and other technical performance metrics. But in practice, it is common for model I/O validation steps to be added later, when scaling an AI product.

Management 244

Management Metrics Machine Learning Modeling 244

Occam's Razor

JULY 6, 2016

Yes, I worry that Analysts, and Marketers, are spending too much time with their head buried in custom reports and advance segments and smart calculated metrics and strategic or tactical dashboards. Today's post is an adjacent mistake: The cardinal sin of spending too much time with data and in reports! The higher order bit is simple.

Testing 86

Testing Marketing Experimentation Cost-Benefit 86

Occam's Razor

DECEMBER 9, 2009

In this blog post I'll share a unique "case study", more like one person's problem, and my advice to them about how to think about the organization problem. If your overall web business is misaligned from an org perspective then honestly there is no hope for you, regardless of where analytics sits.]. You are out man!

Analytics 95

Analytics Data-driven Marketing Sales 95

CIO Business Intelligence

JULY 11, 2023

In this blog post, we will delve deeper into each of these principles and provide concrete examples to illustrate their importance. For example, a good result in a single clinical trial may be enough to consider an experimental treatment or follow-on trial but not enough to change the standard of care for all patients with a specific disease.

Modeling 95

Modeling Testing Experimentation Risk Management 95

DataKitchen

APRIL 13, 2021

Read the complete blog below for a more detailed description of the vendors and their capabilities. A complete DataOps program will have a unified, system-wide view of process metrics using a common data store. Download the 2021 DataOps Vendor Landscape here. DataOps is a hot topic in 2021.

Testing 307

Testing Machine Learning Consulting Data Quality 307

The Unofficial Google Data Science Blog

APRIL 16, 2019

by MICHAEL FORTE Large-scale live experimentation is a big part of online product development. In fact, this blog has published posts on this very topic. This means a small and growing product has to use experimentation differently and very carefully. This blog post is about experimentation in this regime.

Experimentation 68

Experimentation Sales Metrics Measurement 68

DataRobot Blog

SEPTEMBER 30, 2022

By tracking service, drift, prediction data, training data, and custom metrics, you can keep your models and predictions relevant in a fast-changing world. Adoption of AI/ML is maturing from experimentation to deployment. How do you track the integrity of a machine learning model in production? Model Observability can help.

Slice and Dice 52

Slice and Dice Modeling Metrics Experimentation 52

The Unofficial Google Data Science Blog

JUNE 30, 2016

In an ideal world, experimentation through randomization of the treatment assignment allows the identification and consistent estimation of causal effects. We use performance metrics such as bias and mean squared error for the estimation of $delta$, our causal estimand of interest, defined as the average effect of treatment on the treated.

Statistics 44

Statistics Optimization Modeling Experimentation 44

The Unofficial Google Data Science Blog

JULY 31, 2016

This blog post explores how this problem arises in applications at Google, and how we can 'mind our units' through analysis and logging. To figure this out, let's consider an appropriate experimental design. In other words, the teacher is our second kind of unit, the unit of experimentation. However, this is not always the case.

Experimentation 42

Experimentation Testing Measurement Metrics 42

Sisense

JULY 1, 2021

We know in marketing that one of the most powerful ideas is experimentation,” Scott told Sisense. What has held that back is that the gap between idea and implementation has been a real bottleneck for how much experimentation can happen.”. In a world of rapid technological change, digital tools for marketers are having a moment.

Marketing 52

Marketing Experimentation Marketing Analytics ROI 52

The Unofficial Google Data Science Blog

FEBRUARY 29, 2016

Despite a very large number of experimental units, the experiments conducted by LSOS cannot presume statistical significance of all effects they deem practically significant. The result is that experimenters can’t afford to be sloppy about quantifying uncertainty. In statistics, such segments are often called “blocks” or “strata”.

Experimentation 40

Experimentation Metrics Statistics Measurement 40

Occam's Razor

SEPTEMBER 30, 2013

We'll start with digital at the highest strategic level, which leads us into content marketing, from there it is a quick hop over to the challenge of metrics and silos, followed by a recommendation to optimize for the global maxima, and we end with the last two visuals that cover social investment and social content strategy.

Marketing Analytics 154

Marketing Analytics Marketing Visualization Analytics 154

The Unofficial Google Data Science Blog

DECEMBER 14, 2021

This blog post introduces the notions of representational uncertainty and interventional uncertainty to paint a fuller picture of what the practicing data scientist is up against. This blog post does not discuss objectives. But there are other kinds of uncertainty, at least as important, that are not statistical in nature. Finland).

Uncertainty 56

Uncertainty Statistics Measurement Cost-Benefit 56

Sisense

MAY 18, 2021

When you discover data that means something, you need to be agile enough to make experimental changes.”. Mistake #3: Making vanity metrics your main event. When the primary focus becomes likes, retweets, follows, and comments — also known as vanity metrics — marketing efforts become less meaningful for your long-term goals.

Marketing 52

Marketing Data Analytics Analytics Metrics 52