How to plan a human data pilot that tells you something

By the One6 team · October 16, 2026 · 6 min read

A pilot is the cheapest point at which to discover that a dataset will not work. Run well, it tests the specification, the guidelines, the contributors and the quality process all at once. Run badly, it produces a sample that looks fine and proves nothing. This guide covers how to plan one that gives you a clear decision.

1. Decide what the pilot must prove

Write down the question the pilot answers before anything else. "Can this provider produce data we would put in our training set?" is a good question. "Let's see what comes back" is not.

The model behavior the data is meant to improve or measure
What a passing item looks like, and what makes an item fail
The minimum quality level that would justify scaling
Who on your team will judge the output, and how much time they have

2. Write the specification first

The pilot is only as good as the brief behind it. A short written specification removes most of the ambiguity that otherwise shows up as rework.

Data format, modality and required metadata
Contributor profile: languages, expertise, demographics
Guidelines, with worked examples of good and bad items
Acceptance criteria and how they will be checked

3. Size it to answer the question

A pilot should be large enough to show the range of the data, including the hard cases, but small enough that you can review a meaningful share of it yourself. Include the edge cases you care most about, not just typical items, and make sure every contributor profile you plan to use at scale is represented.

4. Agree how quality will be measured

Decide in advance which checks apply, so that the pilot is judged against a standard rather than an impression.

Gold-standard items with known answers
Reviewer sampling of contributor output
Agreement between annotators or raters on the same items
Your own team's review of a random sample

5. Treat disagreements as findings

When contributors disagree, or when your reviewers reject items, look for the cause before blaming the people. Most pilot problems trace back to an unclear guideline or a missing example. Fixing those is the main value of a pilot, and it is far cheaper now than at volume.

6. Make a clear decision

End the pilot with an explicit outcome: scale as planned, scale with changes to the guidelines or contributor profile, run a second pilot, or stop. Write down what changes before production starts, so the lessons carry into the next batch.

A pilot planning checklist

The question the pilot answers, in one sentence
A written specification with acceptance criteria
Guidelines with worked examples
A sample that includes hard cases and every contributor profile
Agreed quality checks and who reviews the output
A decision date and the possible outcomes