Insight
AI Data Annotation Vendor Due Diligence Checklist
A suggested weighted scorecard and document list for evaluating expert data annotation vendors. Labeled as a framework, not a standard.
Published 2026-08-26 · Written by the Stratum editorial team. No individual author or outside clinical or legal reviewer is named for this article.
Due diligence for an expert data vendor is closer to evaluating a specialized services firm than a crowd platform. The useful evidence is how people are sourced and qualified for your domain.
Suggested scorecard
Weights below are a suggested internal framework, not an industry standard. Use the interactive vendor scorecard if you want to score a live conversation.
| Criterion | Weight | Ask |
|---|---|---|
| Domain expertise | 25% | Can the vendor staff the exact profession, specialty, and seniority in the brief? |
| Qualification process | 15% | Do tests resemble production artifacts, with a written pass bar? |
| Quality assurance | 15% | Are calibration, gold sets, and sampling defined before volume starts? |
| Adjudication | 10% | Who resolves disagreement, and are they more senior or more specialized? |
| Security | 10% | Can reviewers work in the client environment with project-scoped access? |
| Capacity | 10% | Are hours from qualified people, not a contractual adjective? |
| Workforce continuity | 5% | What happens when an expert drops in week two? |
| Reporting | 5% | Can you see throughput, vacancy, agreement, and escalations? |
| Commercial terms | 5% | Are production, review, adjudication, and management priced transparently? |
Documents worth requesting
- A redacted qualification task that resembles your artifacts.
- The replacement process, including re-qualification.
- How access is granted and revoked.
- A sample weekly operating report with no fake metrics required—empty is honest if no program has run yet.
Continue with how to choose an outsourcing company.