How to choose a human data provider for AI training and evaluation
By the One6 team · · 7 min read
Most problems with outsourced AI data don't show up in the sales call. They show up three weeks into production, when labels drift, contributors don't match the brief, or nobody can explain why a batch failed review. The questions below are designed to surface those problems before you sign anything.
1. Start with your own specification
Before you compare vendors, write down what a good data point looks like. A provider can only be measured against a standard, and a vague brief produces vague data from every vendor.
- The model behavior you are trying to change, and how you'll measure it
- Format, modality and metadata for each item
- Who should produce the data: native speakers, domain experts, a demographic mix, or trained generalists
- Acceptance criteria: what makes an item fail review
- Volume and timeline for the pilot and for production
2. Ask who will actually do the work
"Global workforce" means very little on its own. What matters is whether the specific people on your project match the brief and how they were selected.
- How are contributors recruited and screened for this kind of task?
- For expert work, how is expertise verified, and can you see anonymized reviewer profiles?
- For language work, are contributors native speakers of the exact regional variety you need?
- What happens when a contributor consistently underperforms?
3. Understand the quality process, not just the quality claim
Every vendor will say their data is high quality. Ask them to describe the mechanism instead.
- Are contributors calibrated on shared examples before production starts?
- How are gold-standard items, reviewer sampling and inter-annotator agreement used?
- Who reviews the reviewers?
- How are disagreements and guideline ambiguities escalated back to your team?
- What quality reporting do you receive with each batch?
4. Insist on a paid pilot
A small pilot is the cheapest way to learn whether a provider can deliver. It tests the guidelines as much as the vendor: most pilots surface ambiguities in the brief that would have been expensive at scale.
Agree in advance what the pilot must demonstrate, how you'll evaluate it, and what happens next if it passes. A provider that resists a pilot, or wants to skip straight to volume, is telling you something.
5. Check data rights, consent and security
- Who owns the delivered data, and are there any restrictions on how you can use it?
- For voice, image or personal data: how is contributor consent obtained, and does it cover your intended use?
- Can the work run under NDA, and how is access to your materials limited?
- Which certifications or audits can the provider actually evidence, as opposed to describe?
6. Match the provider to the type of work
Human data covers very different jobs. High-volume labeling rewards tooling and throughput. Expert evaluation rewards depth and reviewer calibration. Multilingual and speech work depend on genuine native-speaker coverage. Many teams end up using different providers for different workloads — that's normal.
7. Make pricing comparable
Per-item, per-hour and per-project pricing are hard to compare directly. Ask each vendor to price the same pilot specification, and ask what is included: guideline development, QA review, re-work of rejected items, and project management are often where quotes differ.
A short checklist to send vendors
- Describe the contributor profile you would staff for this spec.
- Describe your QA process and what we'll see in each batch report.
- Propose a pilot: size, timeline, and success criteria.
- Explain data ownership, consent and confidentiality terms.
- Price the pilot and an indicative production volume, itemizing what's included.