Insight
How to Qualify Domain Experts for AI Evaluation
A seven-step path from credential check to work-sample qualification, calibration, and re-qualification.
Published 2026-08-26 · Written by the Stratum editorial team. No individual author or outside clinical or legal reviewer is named for this article.
Qualifying domain experts for AI evaluation means proving they can apply your rubric to your artifacts, not only that a credential exists. Resume screening is an entry filter. Qualification is a work sample.
A seven-step qualification path
- Write the expert profile and the pass bar before sending a test.
- Verify the credentials the project actually depends on.
- Send items that look like production, including at least one hard case.
- Score against the rubric, not against “sounds smart.”
- Calibrate survivors on gold examples and edge cases.
- Allow “outside my specialty” as a valid response.
- Re-qualify after a long pause or a rubric version change.
Quality is treated as a cycle, not a single inspection step at the end of a project.
- 01
Qualification
- 02
Calibration
- 03
Production
- 04
Review
- 05
Disagreement detection
- 06
Adjudication
- 07
Feedback
- 08
Recalibration
What a good test includes
At least one item a generalist would pass and a professional would fail. If every candidate scores perfectly, the test is too easy or the key is leaking.
Related: delivery process and how to run a pilot.