article thumbnail

Measuring Validity and Reliability of Human Ratings

The Unofficial Google Data Science Blog

Editor's note : The relationship between reliability and validity are somewhat analogous to that between the notions of statistical uncertainty and representational uncertainty introduced in an earlier post. If they roll two dice and apply a label if the dice rolls sum to 12 they will agree 85% of the time, purely by chance.