Measuring Validity and Reliability of Human Ratings
The Unofficial Google Data Science Blog
JULY 18, 2023
Reliability and validity: Definitions and measurement Two fundamental concepts emerged from this research: reliability and validity. If they roll two dice and apply a label if the dice rolls sum to 12 they will agree 85% of the time, purely by chance. The raw agreement will be (⅚ * ⅚ + ⅙ * ⅙) = 72%.
Let's personalize your content