The Unofficial Google Data Science Blog

APRIL 23, 2024

the weight given to Likes in our video recommendation algorithm) while $Y$ is a vector of outcome measures such as different metrics of user experience (e.g., This blog post discusses such a comprehensive approach that is used at Youtube. the fraction of video recommendations resulted in positive user experiences).

Experimentation 52

Experimentation Optimization Uncertainty Metrics 52

Cloudera

APRIL 9, 2021

This is part 4 in this blog series. This blog series follows the manufacturing and operations data lifecycle stages of an electric car manufacturer – typically experienced in large, data-driven manufacturing companies. The second blog dealt with creating and managing Data Enrichment pipelines. A/B testing).

Machine Learning 98

Machine Learning Forecasting Manufacturing Predictive Analytics 98

Sisense

JANUARY 21, 2020

One of the simplest ways to start exploring your data is to aggregate the metrics you are interested in by their relevant dimensions. A new drug promising to reduce the risk of heart attack was tested with two groups. Everyone wants to get more out of their data, but how exactly to do that can leave you scratching your head.

Testing 104

Testing Risk Data-driven Statistics 104

The Unofficial Google Data Science Blog

OCTOBER 7, 2015

A/B testing is used widely in information technology companies to guide product development and improvements. For questions as disparate as website design and UI, prediction algorithms, or user flows within apps, live traffic tests help developers understand what works well for users and the business, and what doesn’t.

Modeling 61

Modeling Experimentation Knowledge Discovery Testing 61

Occam's Razor

AUGUST 22, 2011

Here you'll find all my blog posts categorized into a structure that will hopefully make it easy for you to discover new content, find answers to your questions, or simply wallow in some excellent analytics narratives. " ~ Web Metrics: "What is a KPI? " + Standard Metrics Revisited Series. How to focus?"

KPI 124

KPI Metrics Measurement ROI 124

Occam's Razor

MAY 15, 2006

There are many types of qualitative data at your disposal including brand buzz, customer satisfaction, net promoter indices, visitor engagement, stickiness, blog-pulse, etc. There is a lot of "buzz" around "buzzy" metrics such as brand value / brand impact, blog-pulse , to name a couple.

Metrics 100

Metrics Experimentation Testing Interactive 100

The Unofficial Google Data Science Blog

JANUARY 14, 2016

Unlike experimentation in some other areas, LSOS experiments present a surprising challenge to statisticians — even though we operate in the realm of “big data”, the statistical uncertainty in our experiments can be substantial. We must therefore maintain statistical rigor in quantifying experimental uncertainty.

Experimentation 41

Experimentation Statistics Metrics Measurement 41

Domino Data Lab

AUGUST 6, 2021

Develop: includes accessing and preparing data and algorithms, researching and development of models and experimentation. Monitor: includes monitoring the performance of the model, tracking metrics, as well as driving adoption of the model by those it was intended to serve.

Data Science 75

Data Science Testing Modeling Publishing 75

DataRobot Blog

JANUARY 10, 2023

ML model builders spend a ton of time running multiple experiments in a data science notebook environment before moving the well-tested and robust models from those experiments to a secure, production-grade environment for general consumption. Capabilities Beyond Classic Jupyter for End-to-end Experimentation. Auto-scale compute.

Experimentation 52

Experimentation Machine Learning Data Science Modeling 52

Cloudera

FEBRUARY 16, 2022

They define each stage from data ingest, feature engineering, model building, testing, deployment and validation. Model performance metrics together with input features, predictions and potentially ground truth values, can be tracked over time. The post Of Muffins and Machine Learning Models appeared first on Cloudera Blog.

Machine Learning 81

Machine Learning Modeling Metadata Recreation/Entertainment 81

Occam's Razor

NOVEMBER 13, 2009

I now have several full time jobs, plus this blog, plus speaking around the world, plus a family, plus… so much more. " This blog post is in three parts: The pitch. Chapter 3 The Awesome World of Clickstream Analysis: Metrics. Chapter 7 Failing Faster: Unleashing the Power of Testing and Experimentation.

Analytics 71

Analytics Metrics Measurement Testing 71

O'Reilly on Data

MAY 14, 2020

AI PMs should enter feature development and experimentation phases only after deciding what problem they want to solve as precisely as possible, and placing the problem into one of these categories. Experimentation: It’s just not possible to create a product by building, evaluating, and deploying a single model.

Management 363

Management Experimentation B2B Machine Learning 363

Occam's Razor

JULY 6, 2016

Yes, I worry that Analysts, and Marketers, are spending too much time with their head buried in custom reports and advance segments and smart calculated metrics and strategic or tactical dashboards. Lab usability testing and online surveys both provide great strategies to obsess about user centric design. No testing.

Testing 86

Testing Marketing Experimentation Cost-Benefit 86

DataKitchen

APRIL 13, 2021

Read the complete blog below for a more detailed description of the vendors and their capabilities. Testing and Data Observability. It orchestrates complex pipelines, toolchains, and tests across teams, locations, and data centers. Testing and Data Observability. Production Monitoring and Development Testing.

Testing 300

Testing Machine Learning Consulting Data Quality 300

Occam's Razor

FEBRUARY 26, 2008

" Or " I proposed testing / surveys / competitive intelligence / Analysts but I was shot down." 1: Implement a Experimentation & Testing Program. # 1: Implement a Experimentation & Testing Program. Here is data from our latest test." And now you have no excuse to avoid testing.

Management 68

Management Experimentation Testing Strategy 68

datapine

MAY 20, 2020

To inspire your customer relationship management report for managing your metrics, explore our cutting-edge selection of KPI examples. When we say “optimal design,” we don’t mean cramming piles of information into one space or being overly experimental with colors. Test, tweak, evolve. Work through your narrative.

Dashboards 265

Dashboards Reporting KPI Visualization 265

O'Reilly on Data

OCTOBER 13, 2020

In Bringing an AI Product to Market , we distinguished the debugging phase of product development from pre-deployment evaluation and testing. During testing and evaluation, application performance is important, but not critical to success. Debugging AI Products. Inference Task Speed and SLOs.

Management 244

Management Metrics Machine Learning Modeling 244

datapine

FEBRUARY 4, 2020

As we have already talked about in our previous blog post on sales reports for daily, weekly or monthly reporting, you need to figure out a couple of things when launching and executing a marketing campaign: are your efforts paying off? Structure your metrics. That way you can choose the best possible metrics for your case.

Reporting 280

Reporting Marketing Advertising Metrics 280

Occam's Razor

JUNE 6, 2007

It has been such an amazing journey to write the book, and for it to come up almost exactly a year after I started this blog. Bonus: Interactive CD: Contains six podcasts, one video, two web analytics metrics definitions documents and five insightful powerpoint presentations. What ideas to test first on your site?

Analytics 60

Analytics Metrics Marketing Dashboards 60

DataRobot Blog

MARCH 16, 2023

This means they need the tools that can help with testing and documenting the model, automation across the entire pipeline and they need to be able to seamlessly integrate the model into business critical applications or workflows. blog series and deep dive into the new 9.0 features over the next few weeks.

Experimentation 98

Experimentation Data-driven Modeling Metadata 98

CIO Business Intelligence

JULY 11, 2023

In this blog post, we will delve deeper into each of these principles and provide concrete examples to illustrate their importance. For example, a good result in a single clinical trial may be enough to consider an experimental treatment or follow-on trial but not enough to change the standard of care for all patients with a specific disease.

Modeling 69

Modeling Testing Experimentation Risk Management 69

Occam's Razor

DECEMBER 9, 2009

In this blog post I'll share a unique "case study", more like one person's problem, and my advice to them about how to think about the organization problem. The four pronged real world tested probing and loaded with politics framework to find a home for Web Analytics: 1. Are there standard tools deployed?

Analytics 95

Analytics Data-driven Marketing Sales 95

The Unofficial Google Data Science Blog

JUNE 30, 2016

In an ideal world, experimentation through randomization of the treatment assignment allows the identification and consistent estimation of causal effects. A naïve way to solve this problem would be to compare the proportion of buyers between the exposed and unexposed groups, using a simple test for equality of means.

Statistics 44

Statistics Optimization Modeling Experimentation 44

The Unofficial Google Data Science Blog

APRIL 16, 2019

by MICHAEL FORTE Large-scale live experimentation is a big part of online product development. In fact, this blog has published posts on this very topic. This means a small and growing product has to use experimentation differently and very carefully. This blog post is about experimentation in this regime.

Experimentation 68

Experimentation Sales Metrics Measurement 68

The Unofficial Google Data Science Blog

JULY 31, 2016

This blog post explores how this problem arises in applications at Google, and how we can 'mind our units' through analysis and logging. At the end of the semester, you administer a proficiency test to the students and compare test performance across the groups. However, this is not always the case.

Experimentation 42

Experimentation Testing Measurement Metrics 42

The Unofficial Google Data Science Blog

DECEMBER 14, 2021

This blog post introduces the notions of representational uncertainty and interventional uncertainty to paint a fuller picture of what the practicing data scientist is up against. This blog post does not discuss objectives. Should we use a t-test or a sign test? to run a brief experiment to test the hypothesis.

Uncertainty 56

Uncertainty Statistics Measurement Cost-Benefit 56

The Unofficial Google Data Science Blog

MAY 31, 2016

Similarly, we could test the effectiveness of a search ad compared to showing only organic search results. A geo experiment is an experiment where the experimental units are defined by geographic regions. In other words, iROAS is the slope of a curve of the response metric plotted against the underlying advertising spend.

Advertising 42

Advertising Testing Sales Statistics 42

DataRobot Blog

FEBRUARY 15, 2023

The real estate market changes over time, so it’s important that our model learns from past data and is tested on a time frame from the future. This helps with getting more creative with your experimentation. The MLOps command center gives you a birds-eye view of your model, monitoring key metrics like accuracy and data drift.

Modeling 52

Modeling Machine Learning Cost-Benefit Metrics 52

Occam's Razor

MARCH 23, 2010

In this blog post let me share with you some ground truths from my own humble experience. Because every tool uses its own sweet metrics definitions, cookie rules, session start and end rules and so much more. Usually at least a test. In fact smarter than you would be through any other channel on the planet! Likely not.

Analytics 118

Analytics Consulting Metrics Experimentation 118

Sisense

MAY 18, 2021

However, as Deven states, avoiding data insights and going with your gut is like choosing all the wrong answers on a test despite your professor giving you the right ones. When you discover data that means something, you need to be agile enough to make experimental changes.”. Mistake #3: Making vanity metrics your main event.

Marketing 52

Marketing Data Analytics Analytics Metrics 52

Occam's Razor

AUGUST 13, 2012

Yehoshua I've covered this topic in detail in this blog post: Multi-Channel Attribution: Definitions, Models and a Reality Check. That means: All of these metrics are off. This is exactly why the Page Value metric (in the past called $index value) was created. Hopefully soon! Edward Cowell. "Was

Modeling 124

Modeling Metrics Data Quality Data-driven 124

Cloudera

JUNE 6, 2019

We sometimes refer to this as splitting “dev/test” from “production” workloads, but we can generalize the approach by referring to the overall priority of the workload for the business. Designing a proper network for VPCs extends beyond the scope of this blog post. 3) By workload priority.

Metadata 85

Metadata Data Lake Optimization Strategy 85

The Unofficial Google Data Science Blog

OCTOBER 21, 2015

We’re not going to engage in this debate but in this blog post we do focus on science. The beliefs of this community are always evolving, and the process of thoughtfully generating, testing, refuting and accepting ideas looks a lot like Science. the power grid, a streaming music service, the human body, the weather).

Slice and Dice 45

Slice and Dice Experimentation Data-driven Data Science 45

Domino Data Lab

MAY 8, 2019

Finale Doshi-Velez, Been Kim (2017-02-28) ; see also the Domino blog article about TCAV. If your “performance” metrics are focused on predictive power, then you’ll probably end up with more complex models, and consequently less interpretable ones. Challenges for Transparency ”. Adrian Weller (2017-07-29). “ Riccardo Guidotti, et al.

Machine Learning 105

Machine Learning Data Science Modeling Visualization 105

Occam's Razor

FEBRUARY 7, 2011

After 416,350 words in posts and 845,128 words in comments on this blog, thus far, there is always more to explore, illuminate and share. It is a great way to stay connected to what's most important to you (and keep the blog and its content relevant!). But each keyword gets "credit" for other metrics.

Analytics 94

Analytics Measurement Metrics Predictive Analytics 94

Occam's Razor

OCTOBER 19, 2010

I was reflecting on that recently and thought it was incredible that in all my years of writing this blog I have never written a blog post, not one single one (!!), My goal is to give you a list of tools that I use in my everyday life as a practitioner (you'll see many of them implemented on this blog). Disclosure].

Analytics 135

Analytics Testing Measurement Optimization 135

Domino Data Lab

AUGUST 26, 2019

This control prevents overfitting and reduction in predictive accuracy on new test data. The project also covers building a pipeline for automating ML workflow and stay tuned for additional hyperparamater content on the Domino Data Science blog. algorithm leaf_size metric metric_params n_jobs n_neighbors p weights.

Testing 79

Testing Modeling Machine Learning Metrics 79