O'Reilly on Data

APRIL 19, 2021

Relatively few respondents are using version control for data and models. Tools for versioning data and models are still immature, but they’re critical for making AI results reproducible and reliable. The biggest skills gaps were ML modelers and data scientists (52%), understanding business use cases (49%), and data engineering (42%).

Enterprise 359

Enterprise Modeling Risk Manufacturing 359

Occam's Razor

MAY 22, 2006

This post is a primer on the delightful world of testing and experimentation (A/B, Multivariate, and a new term from me: Experience Testing). Experimentation and testing help us figure out we are wrong, quickly and repeatedly and if you think about it that is a great thing for our customers, and for our employers. Counter claims?

Experimentation 103

Experimentation Testing Optimization Measurement 103

The Unofficial Google Data Science Blog

JANUARY 14, 2016

Unlike experimentation in some other areas, LSOS experiments present a surprising challenge to statisticians — even though we operate in the realm of “big data”, the statistical uncertainty in our experiments can be substantial. We must therefore maintain statistical rigor in quantifying experimental uncertainty.

Experimentation 41

Experimentation Statistics Metrics Measurement 41

Occam's Razor

AUGUST 22, 2011

" ~ Web Metrics: "What is a KPI? " + Standard Metrics Revisited Series. Key To Your Digital Success: Web Analytics Measurement Model. Web Data Quality: A 6 Step Process To Evolve Your Mental Model. "Engagement" Is Not A Metric, It's An Excuse. Experimentation and Testing: A Primer.

KPI 124

KPI Metrics Measurement ROI 124

Occam's Razor

JULY 24, 2018

work (collection, processing, reporting, analysis), processes, org structure, governance models, last-mile gaps , metrics ladders of awesomeness , and… so… much… more. Remember, tools, work, processes, org structure, governance models, last-mile gaps, metrics ladders of awesomeness, and… so… much… more. The Impact Matrix.

Analytics 93

Analytics Metrics Strategy Measurement 93

CIO Business Intelligence

JUNE 27, 2022

I do still contribute to open-source projects and when I can I try to contribute code to Sentry, but definitely not as much as I want. Tyson: You’re coming up to two years since you added Sentry’s ability to monitor for performance which has some pretty fine-grained metrics in terms of identifying where bottlenecks are located in code.

Software 98

Software Experimentation Testing Marketing 98

CIO Business Intelligence

MAY 1, 2024

Gen AI takes us from single-use models of machine learning (ML) to AI tools that promise to be a platform with uses in many areas, but you still need to validate they’re appropriate for the problems you want solved, and that your users know how to use gen AI effectively. Pilots can offer value beyond just experimentation, of course.

Cost-Benefit 139

Cost-Benefit Metrics Insurance Measurement 139

O'Reilly on Data

MAY 14, 2020

AI PMs should enter feature development and experimentation phases only after deciding what problem they want to solve as precisely as possible, and placing the problem into one of these categories. Experimentation: It’s just not possible to create a product by building, evaluating, and deploying a single model.

Management 363

Management Experimentation B2B Machine Learning 363

Occam's Razor

OCTOBER 21, 2013

I've gone through the five stages in the Kubler-Ross model. You can read this post to learn how these tools collect data: The Definitive Guide To (8) Competitive Intelligence Data Sources. Bonus: For more on next steps and attribution modeling please see: Multi-Channel Attribution Modeling: The Good, Bad and Ugly Models. ].

Optimization 154

Optimization Reporting Experimentation Metrics 154

O'Reilly on Data

OCTOBER 13, 2020

Similarly, in “ Building Machine Learning Powered Applications: Going from Idea to Product ,” Emmanuel Ameisen states: “Indeed, exposing a model to users in production comes with a set of challenges that mirrors the ones that come with debugging a model.”.

Management 253

Management Metrics Machine Learning Modeling 253

datapine

FEBRUARY 4, 2020

But first, we will start with a basic definition and some tips on creating these kinds of reports. Structure your metrics. As with any report you might need to create, structuring and implementing metrics that will tell an interesting and educational data-story is crucial in our digital age. Let’s dig deeper. click to enlarge**.

Reporting 280

Reporting Marketing Advertising Metrics 280

Occam's Razor

AUGUST 12, 2013

than multi-channel attribution modeling. By the time you are done with this post you'll have complete knowledge of what's ugly and bad when it comes to attribution modeling. You'll know how to use the good model, even if it is far from perfect. Multi-Channel Attribution Models. Linear Attribution Model.

Modeling 161

Modeling Optimization Marketing Interactive 161

Occam's Razor

JULY 18, 2011

The only requirement is that your mental model (and indeed, company culture) should be solidly rooted in permission marketing. You just have to have the right mental model (see Seth Godin above) and you have to… wait for it… wait for it… measure everything you do! Just to ensure you are executing against your right mental model.

Metrics 137

Metrics Marketing Measurement Cost-Benefit 137

DataKitchen

APRIL 13, 2021

Because it is such a new category, both overly narrow and overly broad definitions of DataOps abound. DataOps needs a directed graph-based workflow that contains all the data access, integration, model and visualization steps in the data analytic production process. Datatron — Automates deployment and monitoring of AI models.

Testing 300

Testing Machine Learning Consulting Data Quality 300

Occam's Razor

DECEMBER 9, 2009

Which physical organizational model will work best for you? My organization redesign plans have recommended either one of the three models. If you are executing on a centralized model be aware of the pros and cons. Ok, only if it's legal where you live. ;)]. Centralized? Decentralized? Something else? Had to be said.

Analytics 95

Analytics Data-driven Marketing Sales 95

Domino Data Lab

JUNE 23, 2019

Skomoroch proposes that managing ML projects are challenging for organizations because shipping ML projects requires an experimental culture that fundamentally changes how many companies approach building and shipping software. Another pattern that I’ve seen in good PMs is that they’re very metric-driven.

Management 110

Management Machine Learning Experimentation Metrics 110

Occam's Razor

AUGUST 13, 2012

Yehoshua I've covered this topic in detail in this blog post: Multi-Channel Attribution: Definitions, Models and a Reality Check. I explain three different models (Online to Store, Across Multiple Devices, Across Digital Channels) and for each I've highlighted: 1. That means: All of these metrics are off.

Modeling 124

Modeling Metrics Data Quality Data-driven 124

Corinium

APRIL 25, 2019

Yes, definitely! Life insurance needs accurate data on consumer health, age and other metrics of risk. This is a much more proactive and scalable model. It definitely depends on the type of data, no one method is always better than the other. Yes, definitely, the only questions in my mind are when and how it comes about.

Insurance 150

Insurance Risk IoT Cost-Benefit 150

The Unofficial Google Data Science Blog

DECEMBER 14, 2021

The DS team needed an explicit definition of video quality. They agreed to base their definition on the evaluation of trained raters. Most agreed that Like Rate of a video on the site was in fact a reasonable proxy for this definition of video content quality. The model may not hold well for such extrapolation.

Uncertainty 56

Uncertainty Statistics Measurement Cost-Benefit 56

Domino Data Lab

MAY 8, 2019

Paco Nathan’s latest article features several emerging threads adjacent to model interpretability. I’ve been out themespotting and this month’s article features several emerging threads adjacent to the interpretability of machine learning models. Machine learning model interpretability. Introduction. 2018-06-21).

Machine Learning 105

Machine Learning Data Science Modeling Visualization 105

Domino Data Lab

FEBRUARY 4, 2019

Spoiler alert: a research field called curiosity-driven learning is emerging at the nexis of experimental cognitive psychology and industry use cases for machine learning, particularly in gaming AI. Ensure a culture that supports a steady process of learning and experimentation. Problems with training ML models efficiently.

Data Science 66

Data Science Experimentation Data-driven Machine Learning 66

The Unofficial Google Data Science Blog

FEBRUARY 29, 2016

Despite a very large number of experimental units, the experiments conducted by LSOS cannot presume statistical significance of all effects they deem practically significant. The result is that experimenters can’t afford to be sloppy about quantifying uncertainty. In statistics, such segments are often called “blocks” or “strata”.

Experimentation 40

Experimentation Metrics Statistics Measurement 40

Occam's Razor

FEBRUARY 7, 2011

But each keyword gets "credit" for other metrics. The best option is to hire a statistician with experience in data modeling and forecasting. Brian Krick: Best way to measure and communicate "available demand" from available channels (social, search, display) for forecast modeling. Please see the advice above.

Analytics 94

Analytics Measurement Metrics Predictive Analytics 94

Occam's Razor

MARCH 23, 2010

Because every tool uses its own sweet metrics definitions, cookie rules, session start and end rules and so much more. If you don't kill 25% of your metrics each year, you are doing something wrong. Why do you think introducing a completely different set of numbers is going to make your life easier? Likely not.

Analytics 118

Analytics Consulting Metrics Experimentation 118

The Unofficial Google Data Science Blog

OCTOBER 21, 2015

By definition, it is hard to disentangle causes and effects in these settings because it is rarely possible to isolate all relevant factors. It is important to make clear distinctions among each of these, and to advance the state of knowledge through concerted observation, modeling and experimentation.

Slice and Dice 45

Slice and Dice Experimentation Data-driven Data Science 45

Occam's Razor

MAY 27, 2015

It is being hyper-conservative when it comes to creativity and experimentation because of quant-issues. Definitely buy my data books! :). This will definitely get me on the wrong side of many thought leaders in the mobile space, indeed even in the digital space. Please don't have a data-first strategy. And innovation.

Marketing Analytics 153

Marketing Analytics Marketing Analytics Advertising 153

Occam's Razor

OCTOBER 19, 2010

Given the definition above, I am a firm believer in Multiplicity. Mongoose Metrics ~ ifbyphone. I know Mongoose Metrics a bit more and have been impressed with their solution and evolution over the last couple of years. Experimentation and Testing Tools [The "Why" – Part 1]. Second Bit Of Context. Multiplicity.

Analytics 135

Analytics Testing Measurement Optimization 135

Occam's Razor

JANUARY 7, 2014

The organization functions off a clearly defined Digital Marketing & Measurement Model. #1. Before we go too deep, let's get a couple of definitions right first. More on the Digital Marketing & Measurement Model, DMMM, in #2 below.). Pick hard metrics to designate as your key performance indicators.

Data-driven 140

Data-driven Consulting Reporting Marketing 140

O'Reilly on Data

MAY 11, 2020

Not all models are created equal, however: they operate on different principles, and impact us as individuals and communities in different ways. To understand the menagerie of models that are fundamentally altering our individual and shared realities, we need to build a typology, a classification of their effects and impacts.

Modeling 195

Modeling Advertising Recreation/Entertainment Metrics 195