O'Reilly on Data

MARCH 31, 2020

This has serious implications for software testing, versioning, deployment, and other core development processes. The need for an experimental culture implies that machine learning is currently better suited to the consumer space than it is to enterprise companies.

Management 332

Management Machine Learning Experimentation Metrics 332

Octopai

OCTOBER 26, 2022

When DataOps principles are implemented within an organization, you see an increase in collaboration, experimentation, deployment speed and data quality. SPC is the continuous testing of the results of automated manufacturing processes. SPC tests can do the same thing for the data flowing through your pipelines.

Data Quality 52

Data Quality Data Analytics Analytics Manufacturing 52

Occam's Razor

MARCH 23, 2021

This is very hard to do, we now have a proven seven-step experimentation process, with one of the coolest algorithms to pick matched-markets (normally the kiss of death of any large-scale geo experiment). The first component is a gloriously scaled global creative pre-testing program. Matched market tests. The slow music.

Analytics 131

Analytics Scorecard Strategy Testing 131

The Unofficial Google Data Science Blog

OCTOBER 7, 2015

A/B testing is used widely in information technology companies to guide product development and improvements. For questions as disparate as website design and UI, prediction algorithms, or user flows within apps, live traffic tests help developers understand what works well for users and the business, and what doesn’t.

Modeling 61

Modeling Experimentation Knowledge Discovery Testing 61

Cloudera

APRIL 9, 2021

To effectively leverage their predictive capabilities and maximize time-to-value these companies need an ML infrastructure that allows them to quickly move models from data pipelines, to experimentation and into the business. A/B testing). model packaging, deployment and serving. model monitoring.

Machine Learning 98

Machine Learning Forecasting Manufacturing Predictive Analytics 98

Occam's Razor

MAY 15, 2006

There is a lot of "buzz" around "buzzy" metrics such as brand value / brand impact, blog-pulse , to name a couple. IMHO these "buzzy" metrics might be a sub optimal use of time/resources if we don't first have a hard core understanding of customer satisfaction and task completion on our websites.

Metrics 100

Metrics Experimentation Testing Interactive 100

CIO Business Intelligence

JUNE 27, 2022

Tyson: You’re coming up to two years since you added Sentry’s ability to monitor for performance which has some pretty fine-grained metrics in terms of identifying where bottlenecks are located in code. We rely heavily on automated testing. Tyson: That belief in your vision when it’s tested—that is tough!

Software 115

Software Experimentation Testing Marketing 115

Occam's Razor

AUGUST 22, 2011

" ~ Web Metrics: "What is a KPI? " + Standard Metrics Revisited Series. "Engagement" Is Not A Metric, It's An Excuse. Defining a "Master Metric", + a Framework to Gain a Competitive Advantage in Web Analytics. Five Reasons And Awesome Testing Ideas. How do I choose well?

KPI 124

KPI Metrics Measurement ROI 124

Occam's Razor

OCTOBER 21, 2013

As an analyst, I was upset that this change would hurt my ability to analyze the effectiveness of my beloved search engine optimization (SEO) efforts – which are really all about finding the right users using optimal content strategies. These changes impact my AdWords spend sub-optimally. What Is Not Going Away.

Optimization 154

Optimization Reporting Experimentation Metrics 154

Cloudera

FEBRUARY 16, 2022

They define each stage from data ingest, feature engineering, model building, testing, deployment and validation. By logging the performance of every combination of search parameters within an experiment, we can choose the optimal set of parameters when building a model. This supports automation, consistency and reproducibility.

Machine Learning 81

Machine Learning Modeling Metadata Recreation/Entertainment 81

DataRobot Blog

JANUARY 10, 2023

ML model builders spend a ton of time running multiple experiments in a data science notebook environment before moving the well-tested and robust models from those experiments to a secure, production-grade environment for general consumption. Capabilities Beyond Classic Jupyter for End-to-end Experimentation. Auto-scale compute.

Experimentation 52

Experimentation Machine Learning Data Science Modeling 52

CIO Business Intelligence

MAY 1, 2024

Pilots can offer value beyond just experimentation, of course. McKinsey reports that industrial design teams using LLM-powered summaries of user research and AI-generated images for ideation and experimentation sometimes see a reduction upward of 70% in product development cycle times. Now nearly half of code suggestions are accepted.

Cost-Benefit 136

Cost-Benefit Metrics Insurance Measurement 136

O'Reilly on Data

MAY 14, 2020

AI PMs should enter feature development and experimentation phases only after deciding what problem they want to solve as precisely as possible, and placing the problem into one of these categories. Experimentation: It’s just not possible to create a product by building, evaluating, and deploying a single model.

Management 363

Management Experimentation B2B Machine Learning 363

DataKitchen

APRIL 13, 2021

Testing and Data Observability. It orchestrates complex pipelines, toolchains, and tests across teams, locations, and data centers. Prefect Technologies — Open-source data engineering platform that builds, tests, and runs data workflows. Testing and Data Observability. Production Monitoring and Development Testing.

Testing 307

Testing Machine Learning Consulting Data Quality 307

Occam's Razor

NOVEMBER 13, 2009

Chapter 2 The Optimal Strategy for Choosing Your Web Analytics Soul Mate. Chapter 3 The Awesome World of Clickstream Analysis: Metrics. The second half shows exactly how to pick the best metrics for your org and, my absolute favorite (Page 64), how to diagnose the root cause of a metrics performance. A good thing.

Analytics 71

Analytics Metrics Measurement Testing 71

Occam's Razor

FEBRUARY 26, 2008

" Or " I proposed testing / surveys / competitive intelligence / Analysts but I was shot down." 1: Implement a Experimentation & Testing Program. # 1: Implement a Experimentation & Testing Program. Here is data from our latest test." And now you have no excuse to avoid testing.

Management 68

Management Experimentation Testing Strategy 68

datapine

MAY 20, 2020

With a powerful dashboard maker , each point of your customer relations can be optimized to maximize your performance while bringing various additional benefits to the picture. To inspire your customer relationship management report for managing your metrics, explore our cutting-edge selection of KPI examples. Test, tweak, evolve.

Dashboards 265

Dashboards Reporting KPI Visualization 265

Occam's Razor

DECEMBER 9, 2009

The four pronged real world tested probing and loaded with politics framework to find a home for Web Analytics: 1. some reports going out on schedule (even if data pukes) then an optimal path might be to centralize some where (see item #2 below). [B] How long has the company been doing web analytics, what is the landscape of tools?

Analytics 95

Analytics Data-driven Marketing Sales 95

Occam's Razor

SEPTEMBER 15, 2014

Company UX leaders are happy to stink less by taking the sub-optimal path of responsive design, rather than create a mobile-unique experience (your customers tend to do different things on your desktop site than your mobile site!). Media-Mix Modeling/Experimentation. Many reasons. CEOs still don't get it.

Metrics 141

Metrics Reporting Analytics Marketing 141

CIO Business Intelligence

MARCH 5, 2024

Many other platforms, such as Coveo’s Relative Generative Answering , Quickbase AI , and LaunchDarkly’s Product Experimentation , have embedded virtual assistant capabilities but don’t brand them copilots. Today, top AI-assistant capabilities delivering results include generating code, test cases, and documentation.

Cost-Benefit 124

Cost-Benefit Advertising Experimentation Software 124

Occam's Razor

JULY 18, 2011

To not have it as an active part of your marketing portfolio is sub-optimal. You must use metrics that are unique to the medium. Ready for the best email marketing campaign metrics? So for our email campaign analysis let’s look at metrics using that framework. Optimal Acquisition Email Metrics. Outcomes.

Metrics 137

Metrics Marketing Measurement Cost-Benefit 137

The Unofficial Google Data Science Blog

JUNE 30, 2016

In an ideal world, experimentation through randomization of the treatment assignment allows the identification and consistent estimation of causal effects. A naïve way to solve this problem would be to compare the proportion of buyers between the exposed and unexposed groups, using a simple test for equality of means.

Statistics 44

Statistics Optimization Modeling Experimentation 44

datapine

FEBRUARY 4, 2020

Structure your metrics. As with any report you might need to create, structuring and implementing metrics that will tell an interesting and educational data-story is crucial in our digital age. That way you can choose the best possible metrics for your case. Regularly monitor your data. 1) Marketing CMO report.

Reporting 280

Reporting Marketing Advertising Metrics 280

Occam's Razor

NOVEMBER 12, 2006

And no that is not Visits or Total Visitors in the denominator, for my perspective on why that is sub-optimal click here and read the part about Why use Unique Visitors.]. The problem is that you are there just to look at the car, maybe take it for a test drive. Is Real Conversion Rate metric a good one? What do you think?

Measurement 99

Measurement Analytics Optimization Marketing 99

Occam's Razor

JUNE 6, 2007

Bonus: Interactive CD: Contains six podcasts, one video, two web analytics metrics definitions documents and five insightful powerpoint presentations. Experimentation & Testing (A/B, Multivariate, you name it). What's the optimal organization structure (and who should own web analytics!)? Clicks and outcomes.

Analytics 60

Analytics Metrics Marketing Dashboards 60

Occam's Razor

NOVEMBER 17, 2008

There is a tendency to think experimentation and testing is optional. You can start for free with a superb tool: Google's Website Optimizer. For example I am quite fond of the fact that with Offermatica you can "trigger" tests based on behavior. I cannot recommend enough the wisdom of starting with a A/B test.

Testing 112

Testing Experimentation Risk Optimization 112

Domino Data Lab

JUNE 23, 2019

Skomoroch proposes that managing ML projects are challenging for organizations because shipping ML projects requires an experimental culture that fundamentally changes how many companies approach building and shipping software. Another pattern that I’ve seen in good PMs is that they’re very metric-driven. Transcript.

Management 110

Management Machine Learning Experimentation Metrics 110

The Unofficial Google Data Science Blog

APRIL 16, 2019

by MICHAEL FORTE Large-scale live experimentation is a big part of online product development. This means a small and growing product has to use experimentation differently and very carefully. This blog post is about experimentation in this regime. Such decisions involve an actual hypothesis test on specific metrics (e.g.

Experimentation 68

Experimentation Sales Metrics Measurement 68

AWS Big Data

JULY 26, 2023

The vector engine supports the popular distance metrics such as Euclidean, cosine similarity, and dot product, and can accommodate 16,000 dimensions, making it well-suited to support a wide range of foundational and other AI/ML models. To create the vector index, you must define the vector field name, dimensions, and the distance metric.

Metadata 52

Metadata Cost-Benefit Testing Metrics 52

Occam's Razor

AUGUST 12, 2013

The outcome in either scenario is a restructuring of the organization that is exquisitely geared towards taking advantage of portfolio optimization. You should not treat your marketing optimization program with the same level of outcome optimization that is applied to five-year-olds. From a Venn -diagram. Who would have thunk?].

Modeling 161

Modeling Optimization Marketing Interactive 161

Cloudera

JUNE 6, 2019

This strategy works well for managing internal chargebacks, limiting the impact of less sophisticated users on more experienced users, and overall encouraging individuals to think about and optimize their jobs and queries now that they have a smaller (but dedicated) cluster. 2) By workload type. 3) By workload priority.

Metadata 85

Metadata Data Lake Optimization Strategy 85

Occam's Razor

MARCH 23, 2010

My problem with these mistruths and FUD is that they result in a ton of practitioners and companies making profoundly sub optimal choices, which in turn results in not just much longer slogs but also spectacular career implosions and the entire web analytics industry suffering. Usually at least a test. This is sad. Likely not.

Analytics 118

Analytics Consulting Metrics Experimentation 118

Occam's Razor

AUGUST 13, 2012

That means: All of these metrics are off. If you want to make the smartest decisions about your budget allocation then leveraging the time tested methodology of media mix modeling (at its core powered by controlled experiments) is the only way to go. I can use that to hypothesize what an optimal budget allocation might look like.

Modeling 124

Modeling Metrics Data Quality Data-driven 124

Domino Data Lab

MAY 8, 2019

If your “performance” metrics are focused on predictive power, then you’ll probably end up with more complex models, and consequently less interpretable ones. They also require advanced skills in statistics, experimental design, causal inference, and so on – more than most data science teams will have.

Machine Learning 105

Machine Learning Data Science Modeling Visualization 105

Occam's Razor

AUGUST 23, 2010

Why comes from lab usability studies , website surveys , "follow me home" exercises, experimentation & testing , and other such delightful endeavors. I know that you even realize Why is ever easier to accomplish (usability studies are economical, surveys and testing platforms start at the sweet price of free!).

Analytics 116

Analytics Cost-Benefit Testing Optimization 116

Domino Data Lab

AUGUST 26, 2019

This control prevents overfitting and reduction in predictive accuracy on new test data. When hyperparameters are also used for optimization and tuned for a specific dataset, they also impact predictive accuracy. Searching is a very general approach to solving or optimizing problems. We need some sort of well-defined process?

Testing 79

Testing Modeling Machine Learning Metrics 79

Occam's Razor

OCTOBER 19, 2010

If you have evolved to a stage that you need behavior targeting then get Omniture Test and Target or Sitespect. A huge vast majority of clicks coming from search engines continue to be organic clicks (which is why I love and adore search engine optimization). Mongoose Metrics ~ ifbyphone. Google Website Optimizer.

Analytics 135

Analytics Testing Measurement Optimization 135