Definition
What is inter-annotator agreement
Inter-annotator agreement is a measure of how often two or more reviewers assign the same label to the same item. Simple percent agreement is a starting diagnostic; chance-corrected coefficients exist for more formal analysis.
What is inter-annotator agreement?
Inter-annotator agreement is a measure of how often two or more reviewers assign the same label to the same item. Simple percent agreement is a starting diagnostic; chance-corrected coefficients exist for more formal analysis.
Why it matters
Low agreement can mean a weak rubric, a mixed cohort, or a genuinely hard item. High agreement on trivial items is not evidence of expert quality.
Example
Dual review on a 50-item calibration set, with disagreements sent to adjudication rather than forced consensus.
When it is used
You need to watch quality after calibration, after a rubric change, or when new reviewers join.
Common mistakes
- Treating any disagreement as incompetence
- Reporting agreement without describing the task difficulty
- Using chance-corrected statistics without stating the formula
What kind of experts do you need?
Share the profession, specialty, experience, location, headcount, hours, duration, and project description. We assemble the expert capacity.
- Need 25 licensed nurses for a clinical AI evaluation
- Need 15 attorneys in a specific practice area
- Need 20 PhD scientists for benchmark creation
- Need 30 senior software engineers for code evaluation