Domino Data Lab

NOVEMBER 14, 2019

This Domino Data Science Field Note covers a proposed definition of interpretability and distilled overview of the PDR framework. James Murdoch, Chandan Singh, Karl Kumber, and Reza Abbasi-Asi’s recent paper, “Definitions, methods, and applications in interpretable machine learning” Introduction. Yet, Bin Yu, W.

Modeling 71

Modeling Machine Learning Data Science Experimentation 71

Data Science and Beyond

APRIL 5, 2021

The initial covid-19 lockdown provided me with extra free time to make the measurement and offsetting of Automattic’s emissions from data centre power use happen. My main "day job" focus in 2020 was on being the tech lead for Automattic’s new experimentation platform (ExPlat). Technical work.

Experimentation 61

Experimentation Testing Publishing Measurement 61

Domino Data Lab

AUGUST 6, 2021

The early definition now accepted as standard in the industry is: “The use of models invariably presents model risk, which is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports.” The genesis of Model Risk Management was in the banking industry. Model implementation.

Risk Management 64

Risk Management Risk Modeling Enterprise 64

Occam's Razor

AUGUST 22, 2011

Key To Your Digital Success: Web Analytics Measurement Model. " Measuring Incrementality: Controlled Experiments to the Rescue! Barriers To An Effective Web Measurement Strategy [+ Solutions!]. Measuring Online Engagement: What Role Does Web Analytics Play? "Engagement" How Do I Measure Success?

KPI 124

KPI Metrics Measurement ROI 124

Occam's Razor

APRIL 8, 2013

Sometimes, we escape the clutches of this sub optimal existence and do pick good metrics or engage in simple A/B testing. First, you figure out what you want to improve; then you create an experiment; then you run the experiment; then you measure the results and decide what to do. Testing out a new feature. Form a hypothesis.

Metrics 156

Metrics KPI Analytics Key Performance Indicator 156

The Unofficial Google Data Science Blog

JANUARY 14, 2016

Unlike experimentation in some other areas, LSOS experiments present a surprising challenge to statisticians — even though we operate in the realm of “big data”, the statistical uncertainty in our experiments can be substantial. We must therefore maintain statistical rigor in quantifying experimental uncertainty.

Experimentation 41

Experimentation Statistics Metrics Measurement 41

CIO Business Intelligence

MAY 1, 2024

Pilots can offer value beyond just experimentation, of course. McKinsey reports that industrial design teams using LLM-powered summaries of user research and AI-generated images for ideation and experimentation sometimes see a reduction upward of 70% in product development cycle times. What are you measuring?

Cost-Benefit 143

Cost-Benefit Metrics Insurance Measurement 143

Ontotext

APRIL 3, 2024

So, this is a big driver for the outcome because when you are saving money for the business, you can measure it and see its value. Nimit Mehta: I think that 2024 is going to be a buckle-down year, but, at the same time, we’ll see a rapid explosion of experimentation. That will definitely stick. It’s open. Especially now.

Technology 59

Technology Risk Digital Transformation Measurement 59

Domino Data Lab

OCTOBER 2, 2019

As data science work is experimental and probabilistic in nature, data scientists are often faced with making inferences. The case that might be familiar to you is an AB test. You can make a change to a product and test it against the original version of the product. A complementary Domino project is available. . Introduction.

Machine Learning 68

Machine Learning Measurement Modeling Testing 68

Occam's Razor

NOVEMBER 12, 2006

Mostly because short term goals drive a lot of what we do and if you are selling something on your website then it only seems to make logical sense that we measure conversion rate and get it up as high as we can as fast as we can. So measure Bounce Rate of your website. Another reason is that we over estimate what is possible.

Measurement 99

Measurement Analytics Optimization Marketing 99

Ontotext

MARCH 29, 2021

More than 10 years ago we even had experimental support of what we called triplesets , which allowed the association of metadata to sets of statements. Vassil Momtchev: We were very pleased to have a strong technical team, like the one of Synaptica, to test drive our RDF-star implementation on a serious task and provide feedback.

Metadata 59

Metadata IT Modeling Experimentation 59

O'Reilly on Data

OCTOBER 19, 2021

While there isn’t an authoritative definition for the term, it shares its ethos with its predecessor, the DevOps movement in software engineering: by adopting well-defined processes, modern tooling, and automated workflows, we can streamline the process of moving from development to robust production deployments. This approach is not novel.

IT 346

IT Testing Experimentation Software 346

O'Reilly on Data

MAY 14, 2020

AI PMs should enter feature development and experimentation phases only after deciding what problem they want to solve as precisely as possible, and placing the problem into one of these categories. Experimentation: It’s just not possible to create a product by building, evaluating, and deploying a single model.

Management 363

Management Experimentation B2B Machine Learning 363

DataKitchen

APRIL 13, 2021

Because it is such a new category, both overly narrow and overly broad definitions of DataOps abound. Testing and Data Observability. It orchestrates complex pipelines, toolchains, and tests across teams, locations, and data centers. Testing and Data Observability. Production Monitoring and Development Testing.

Testing 307

Testing Machine Learning Consulting Data Quality 307

Occam's Razor

FEBRUARY 26, 2008

" Or " I proposed testing / surveys / competitive intelligence / Analysts but I was shot down." 1: Implement a Experimentation & Testing Program. # 1: Implement a Experimentation & Testing Program. Here is data from our latest test." And now you have no excuse to avoid testing.

Management 68

Management Experimentation Testing Strategy 68

O'Reilly on Data

OCTOBER 13, 2020

In Bringing an AI Product to Market , we distinguished the debugging phase of product development from pre-deployment evaluation and testing. This distinction assumes a slightly different definition of debugging than is often used in software development. require not only disclosure, but also monitored testing.

Management 244

Management Metrics Machine Learning Modeling 244

The Unofficial Google Data Science Blog

DECEMBER 14, 2021

In this post, we discuss three types of uncertainty: Statistical uncertainty : the gap between the estimand , the unobserved property of the population we wish to measure, and an estimate of it from observed data. Representational uncertainty : the gap between the desired meaning of some measure and its actual meaning.

Uncertainty 56

Uncertainty Statistics Measurement Cost-Benefit 56

Occam's Razor

OCTOBER 21, 2013

You can read this post to learn how these tools collect data: The Definitive Guide To (8) Competitive Intelligence Data Sources. If you want to stress test this,… go back to your 2011 (pre- not provided ) data for paid and organic and see what you can find. Controlled experimentation. I really loved having this data.

Optimization 154

Optimization Reporting Experimentation Metrics 154

Occam's Razor

JUNE 6, 2007

Bonus: Interactive CD: Contains six podcasts, one video, two web analytics metrics definitions documents and five insightful powerpoint presentations. Experimentation & Testing (A/B, Multivariate, you name it). What ideas to test first on your site? It is a book about Web Analytics 2.0. Qualitative and quantitative.

Analytics 60

Analytics Metrics Marketing Dashboards 60

Domino Data Lab

JUNE 23, 2019

Skomoroch proposes that managing ML projects are challenging for organizations because shipping ML projects requires an experimental culture that fundamentally changes how many companies approach building and shipping software. These measurement-obsessed companies have an advantage when it comes to AI. It is similar to R&D.

Management 110

Management Machine Learning Experimentation Metrics 110

datapine

FEBRUARY 4, 2020

Management thinker Peter Drucker once stated, “if you can’t measure it, you can’t improve it” – and he couldn’t be more right. But first, we will start with a basic definition and some tips on creating these kinds of reports. To know if you are successful, you first need to define success and track it. Let’s dig deeper.

Reporting 280

Reporting Marketing Advertising Metrics 280

Domino Data Lab

MAY 8, 2019

In contrast, some researchers who are exploring the definitions, possibilities, and limitations of model interpretation point toward more comprehensive views. There’s no one-size-fits-all definition of what’s required across all verticals. measure the subjects’ ability to trust the models’ results.

Machine Learning 105

Machine Learning Data Science Modeling Visualization 105

Occam's Razor

JULY 18, 2011

You just have to have the right mental model (see Seth Godin above) and you have to… wait for it… wait for it… measure everything you do! For everything you do it is important to measure your effectiveness of all three phases of your effort: Acquisition. You’re trying to measure how well you are doing to: Send emails.

Metrics 137

Metrics Marketing Measurement Cost-Benefit 137

The Unofficial Google Data Science Blog

OCTOBER 21, 2015

The beliefs of this community are always evolving, and the process of thoughtfully generating, testing, refuting and accepting ideas looks a lot like Science. By definition, it is hard to disentangle causes and effects in these settings because it is rarely possible to isolate all relevant factors.

Slice and Dice 45

Slice and Dice Experimentation Data-driven Data Science 45

Occam's Razor

AUGUST 12, 2013

In a recent post, Multi-Channel Attribution: Definitions, Models and a Reality Check , I outlined three distinct attribution challenges. I strongly encourage you to read the post and deeply understand all three and what your marketing and measurement possibilities and limitations are. . ~ Multi-Channel Attribution Analysis. ~

Modeling 161

Modeling Optimization Marketing Interactive 161

Occam's Razor

MARCH 23, 2010

Too many new things are happening too fast and those of us charged with measuring it have to change the wheels while the bicycle is moving at 30 miles per hour (and this bicycle will become a car before we know it – all while it keeps moving, ever faster). Usually at least a test. And I doubt it is going to happen soon.

Analytics 118

Analytics Consulting Metrics Experimentation 118

Occam's Razor

FEBRUARY 7, 2011

Brian Krick: Best way to measure and communicate "available demand" from available channels (social, search, display) for forecast modeling. Additionally, it is exceptionally difficult to measure available demand because 1. please refer to the controlled experimentation section, page 205, in the book for more.

Analytics 94

Analytics Measurement Metrics Predictive Analytics 94

Occam's Razor

OCTOBER 19, 2010

Given the definition above, I am a firm believer in Multiplicity. If you have evolved to a stage that you need behavior targeting then get Omniture Test and Target or Sitespect. You'll measure Task Completion Rate in 4Q (below). You'll measure Share of Search using Insights for Search (below). Multiplicity.

Analytics 135

Analytics Testing Measurement Optimization 135

Occam's Razor

AUGUST 13, 2012

Yehoshua Coren: Best ways to measure user behavior in a multi-touch, multi-device digital world. Yehoshua I've covered this topic in detail in this blog post: Multi-Channel Attribution: Definitions, Models and a Reality Check. What's possible to measure. What's not possible to measure. Let's do this!

Modeling 124

Modeling Metrics Data Quality Data-driven 124

O'Reilly on Data

DECEMBER 9, 2019

We use the diagnostic test results of our regression model to support the reasons why CIs should not be used in financial data analyses. The probability of an event should be measured empirically by repeating similar experiments ad nauseam —either in reality or hypothetically. Image Source: Wikimedia Commons.

Statistics 27

Statistics Uncertainty Risk Marketing 27

Occam's Razor

JANUARY 7, 2014

The organization functions off a clearly defined Digital Marketing & Measurement Model. #1. Before we go too deep, let's get a couple of definitions right first. More on the Digital Marketing & Measurement Model, DMMM, in #2 below.). Four Useless KPI Measurement Techniques. #9:

Data-driven 140

Data-driven Consulting Reporting Marketing 140

Occam's Razor

NOVEMBER 2, 2015

Higher Order Bits: Human vs. Business, Success KPIs, S-T-D-C Framework, MoR Test. For the rest of this post, I'm going to use the first three to capture the essence of social engagement and brand impact, and one to measure impact on the business. It covers, content, marketing and measurement. It is pronounced the more test.

B2B 167

B2B Metrics Testing Recreation/Entertainment 167