Insight
When AI Training Data Requires Domain Experts
Use the Expert Necessity Test to decide when healthcare, legal, finance, science, engineering, and software data needs professional reviewers.
Published 2026-08-25 · Written by the Stratum editorial team. No individual author or outside clinical or legal reviewer is named for this article.
AI training data requires domain experts when an incorrect answer can look right to a general reviewer. That is the whole test. If your rater would need the same education, license, or years of practice as the user you claim to serve, you are no longer in general annotation.
A task likely requires domain experts when an incorrect answer could appear plausible to a general reviewer but obvious to a professional.
1
Domain knowledge
Does the task require facts or methods a professional would be expected to know?
2
Professional judgment
Would two trained people still need a standard of care, not just a style guide?
3
Error consequence
If a plausible error ships into training or evaluation, what breaks?
4
Ambiguity
Can an answer look correct to a general reviewer while being wrong to a specialist?
5
Need for adjudication
Will qualified reviewers disagree often enough that a senior expert must resolve the label?
| Dimension | Low | Medium | High |
|---|---|---|---|
| Domain knowledge | Generalist can judge | Mixed or specialized content | Professional standard required |
| Professional judgment | Generalist can judge | Mixed or specialized content | Professional standard required |
| Error consequence | Generalist can judge | Mixed or specialized content | Professional standard required |
| Ambiguity | Generalist can judge | Mixed or specialized content | Professional standard required |
| Need for adjudication | Generalist can judge | Mixed or specialized content | Professional standard required |
This is a qualitative planning aid, not a statistically validated instrument.
Healthcare
A model can produce a careful-sounding plan that misses a contraindication or a red-flag symptom. A general rater hears caution. A clinician hears an unsafe omission. Medical data annotation is therefore a clinical judgment task. See medical data annotation.
Law
A memo can cite a real case for a proposition the case does not hold, or apply the wrong jurisdiction’s default rule. The prose still looks like law. Attorneys notice. That is why legal AI evaluation is specified by practice and governing law. See legal data annotation.
Finance
Spreadsheets forgive fluency. Double-counted cash, misclassified extras, and silently optimistic comparables survive a visual scan. Finance professionals are there to reject tidy errors. See financial data annotation.
Science
Citation laundering and method collapse are the characteristic failures. The paper exists. The claim does not. Researchers are hired to notice that gap, not to reward technical vocabulary. See scientific evaluation.
Engineering
A troubleshooting procedure can be internally consistent and still violate a code, a load case, or a physical constraint. Engineers score whether the advice would be allowed to proceed. See engineering data annotation.
Software
Plausible code is the default product of code models. Experienced engineers check compilation, tests, edge cases, architecture, and whether the patch solves the stated ticket. See code evaluation.
Level 1
General annotation
Use when errors are obvious and domain knowledge is not required.
- Classification
- Transcription
- Simple preference tasks
Level 2
Skilled evaluation
Use when the work needs technical literacy or structured reasoning, but not a licensed professional.
- Technical review
- Structured reasoning
- Specialized content
Level 3
Professional expert evaluation
Use when an incorrect answer can look plausible to a generalist and the consequence is material.
- Clinical
- Legal
- Financial
- Scientific
- Advanced engineering
Level 1 data can still be valuable. The mistake is paying Level 1 rates and expecting Level 3 judgment, or the reverse: staffing specialists on tasks a careful generalist could do. Use the test, then write the training-data specification.