O'Reilly on Data

MARCH 31, 2020

All you need to know for now is that machine learning uses statistical techniques to give computer systems the ability to “learn” by being trained on existing data. This has serious implications for software testing, versioning, deployment, and other core development processes. Machine learning adds uncertainty.

Management 332

Management Machine Learning Experimentation Metrics 332

Octopai

OCTOBER 26, 2022

When DataOps principles are implemented within an organization, you see an increase in collaboration, experimentation, deployment speed and data quality. Continuous pipeline monitoring with SPC (statistical process control). SPC is the continuous testing of the results of automated manufacturing processes. Let’s take a look.

Data Quality 52

Data Quality Data Analytics Analytics Manufacturing 52

Occam's Razor

NOVEMBER 17, 2008

There is a tendency to think experimentation and testing is optional. Just don't fall for their bashing of all other vendors or their silly claims, false, of "superiority" in terms of running 19 billion combinations of tests or the bonus feature of helping you into your underwear each morning. And I meant every word of it.

Testing 112

Testing Experimentation Risk Optimization 112

The Unofficial Google Data Science Blog

JANUARY 14, 2016

Unlike experimentation in some other areas, LSOS experiments present a surprising challenge to statisticians — even though we operate in the realm of “big data”, the statistical uncertainty in our experiments can be substantial. We must therefore maintain statistical rigor in quantifying experimental uncertainty.

Experimentation 41

Experimentation Statistics Metrics Measurement 41

Occam's Razor

MARCH 23, 2021

This is very hard to do, we now have a proven seven-step experimentation process, with one of the coolest algorithms to pick matched-markets (normally the kiss of death of any large-scale geo experiment). The first component is a gloriously scaled global creative pre-testing program. Matched market tests. The slow music.

Analytics 131

Analytics Scorecard Strategy Testing 131

The Unofficial Google Data Science Blog

OCTOBER 12, 2017

1]" Statistics, as a discipline, was largely developed in a small data world. With more features come more potential post hoc hypotheses about what is driving metrics of interest, and more opportunity for exploratory analysis. We must correct for multiple hypothesis tests. We ought not dredge our data.

Experimentation 61

Experimentation Testing Statistics Metrics 61

Sisense

JANUARY 21, 2020

One of the simplest ways to start exploring your data is to aggregate the metrics you are interested in by their relevant dimensions. This is an example of Simpon’s paradox , a statistical phenomenon in which a trend that is present when data is put into groups reverses or disappears when the data is combined.

Testing 104

Testing Risk Data-driven Statistics 104

The Unofficial Google Data Science Blog

APRIL 16, 2019

by MICHAEL FORTE Large-scale live experimentation is a big part of online product development. This means a small and growing product has to use experimentation differently and very carefully. This blog post is about experimentation in this regime. Such decisions involve an actual hypothesis test on specific metrics (e.g.

Experimentation 68

Experimentation Sales Metrics Measurement 68

Occam's Razor

AUGUST 22, 2011

" ~ Web Metrics: "What is a KPI? " + Standard Metrics Revisited Series. "Engagement" Is Not A Metric, It's An Excuse. Defining a "Master Metric", + a Framework to Gain a Competitive Advantage in Web Analytics. Five Reasons And Awesome Testing Ideas. How do I choose well?

KPI 124

KPI Metrics Measurement ROI 124

Domino Data Lab

JUNE 23, 2019

Skomoroch proposes that managing ML projects are challenging for organizations because shipping ML projects requires an experimental culture that fundamentally changes how many companies approach building and shipping software. Another pattern that I’ve seen in good PMs is that they’re very metric-driven. Transcript.

Management 110

Management Machine Learning Experimentation Metrics 110

Domino Data Lab

DECEMBER 4, 2020

Having calculated AUC/AUMC, we can further derive a number of useful metrics like: Total clearance of the drug from plasma. Domino Lab supports both interactive and batch experimentation with all popular IDEs and notebooks (Jupyter, RStudio, SAS, Zeppelin, etc.). The area under the first moment curve would respectively be. cl_f = NCA.cl(pain_nca)

Metrics 59

Metrics Data Science Knowledge Discovery Measurement 59

The Unofficial Google Data Science Blog

JULY 31, 2016

At the end of the semester, you administer a proficiency test to the students and compare test performance across the groups. For example, you might compare the average test score among students exposed to the new curriculum to the average score among the controls. Sounds straightforward, right?

Experimentation 42

Experimentation Testing Measurement Metrics 42

The Unofficial Google Data Science Blog

MAY 31, 2016

Similarly, we could test the effectiveness of a search ad compared to showing only organic search results. A geo experiment is an experiment where the experimental units are defined by geographic regions. In other words, iROAS is the slope of a curve of the response metric plotted against the underlying advertising spend.

Advertising 42

Advertising Testing Sales Statistics 42

Domino Data Lab

AUGUST 26, 2019

This control prevents overfitting and reduction in predictive accuracy on new test data. If we don’t like the result of testing after training, we can also select an entirely different learning machine with different internals. algorithm leaf_size metric metric_params n_jobs n_neighbors p weights.

Testing 79

Testing Modeling Machine Learning Metrics 79

Occam's Razor

MARCH 23, 2010

Part of it is fueled by a vocal minority genuinely upset that 10 years on we are still not a statistically powered bunch doing complicated analysis that is shifting paradigms. Because every tool uses its own sweet metrics definitions, cookie rules, session start and end rules and so much more. Usually at least a test.

Analytics 118

Analytics Consulting Metrics Experimentation 118

Insight

MARCH 12, 2020

The most powerful approach for the first task is to use a ‘language model’ (LM), i.e. a statistical model of natural language. I tested several different flavors of BERT for use as synopsis classifiers before settling on the DistilBERT model from Hugging Face. On my test set, this approach resulted in~75–95% accuracy and ~.65

Modeling 59

Modeling Metadata Publishing Sales 59

The Unofficial Google Data Science Blog

MARCH 22, 2018

But what if users don't immediately uptake the new experimental version? Background At Google, experimentation is an invaluable tool for making decisions and inference about new products and features. by DANIEL PERCIVAL Randomized experiments are invaluable in making product decisions, including on mobile apps.

Experimentation 62

Experimentation Measurement Modeling Statistics 62

Domino Data Lab

MAY 8, 2019

If your “performance” metrics are focused on predictive power, then you’ll probably end up with more complex models, and consequently less interpretable ones. They also require advanced skills in statistics, experimental design, causal inference, and so on – more than most data science teams will have.

Machine Learning 105

Machine Learning Data Science Modeling Visualization 105

Domino Data Lab

AUGUST 22, 2019

Although it’s not perfect, [Note: These are statistical approximations, of course!] Note: A test set of 19,500 such analogies was developed by Tomas Mikolov and his colleagues in their 2013 word2vec paper. This test set is available at download.tensorflow.org/data/questions-words.txt.]. Note that the final test word in Table 11.2—ma’am—is

Deep Learning 111

Deep Learning Modeling Metrics Testing 111

The Unofficial Google Data Science Blog

JUNE 30, 2016

In an ideal world, experimentation through randomization of the treatment assignment allows the identification and consistent estimation of causal effects. A naïve way to solve this problem would be to compare the proportion of buyers between the exposed and unexposed groups, using a simple test for equality of means.

Statistics 44

Statistics Optimization Modeling Experimentation 44

The Unofficial Google Data Science Blog

OCTOBER 21, 2015

The beliefs of this community are always evolving, and the process of thoughtfully generating, testing, refuting and accepting ideas looks a lot like Science. It is important to make clear distinctions among each of these, and to advance the state of knowledge through concerted observation, modeling and experimentation.

Slice and Dice 45

Slice and Dice Experimentation Data-driven Data Science 45

Occam's Razor

JUNE 6, 2007

Bonus: Interactive CD: Contains six podcasts, one video, two web analytics metrics definitions documents and five insightful powerpoint presentations. Experimentation & Testing (A/B, Multivariate, you name it). What ideas to test first on your site? Immediately actionable web analytics (your biggest worries covered).

Analytics 60

Analytics Metrics Marketing Dashboards 60

Occam's Razor

JANUARY 9, 2012

If you are doing lame stuff, why try harder in an analytics context by asking for Economic Value or Visitor Loyalty or Conversation Rate or a thousand other super powerful and insightful metrics ? AND you can have analysis of your risk in almost real time to get an early read and in a few days with statistical significance!

Marketing 140

Marketing B2B Measurement Risk 140

IBM Big Data Hub

SEPTEMBER 25, 2023

If the exploratory work needs to move on to testing and production, they can plan appropriately. Presto provides a long list of functions, operators, and expressions as part of its open source offering, including standard functions, maps, arrays, mathematical, and statistical functions. This way, the queries run much faster.

OLAP 90

OLAP Data Lake Data-driven Snapshot 90