Measuring Validity and Reliability of Human Ratings
The Unofficial Google Data Science Blog
JULY 18, 2023
Once we’ve answered that, we will then define and use metrics to understand the quality of human-labeled data, along with a measurement framework that we call Cross-replication Reliability or xRR. If they roll two dice and apply a label if the dice rolls sum to 12 they will agree 85% of the time, purely by chance.
52 
Let's personalize your content