article thumbnail

Fundamentals of Data Mining

Data Science 101

This data alone does not make any sense unless it’s identified to be related in some pattern. Data mining is the process of discovering these patterns among the data and is therefore also known as Knowledge Discovery from Data (KDD). You might be wondering what benefit you can get out of these techniques?

article thumbnail

Variance and significance in large-scale online services

The Unofficial Google Data Science Blog

by AMIR NAJMI Running live experiments on large-scale online services (LSOS) is an important aspect of data science. In this post we explore how and why we can be “ data-rich but information-poor ”. There are many reasons for the recent explosion of data and the resulting rise of data science.

article thumbnail

Using Empirical Bayes to approximate posteriors for large "black box" estimators

The Unofficial Google Data Science Blog

Empirical Bayes methods find a prior such that when we add Poisson noise, we fit the distribution of our observed data. For an introduction to Empirical Bayes, see the paper [3] by Brad Efron (with more in his book [4]). In Figure 2, the red line shows a Gamma prior that leads to a good fit. How exactly should we model $G$?

KDD 40