Practical guides
How to choose an AI training data vendor: 12 questions
Twelve questions that separate serious data vendors from resellers: provenance, consent, QA methodology, pay transparency and what happens when delivery misses spec.
Key takeaways
- Ask to see an actual consent form in the original language. Most vendors cannot produce one.
- Ask what they pay contributors. Reluctance to answer is itself an answer.
- Ask for the failure numbers — rejected batches, quota shortfalls, disagreement rates.
- A vendor who tells you not to buy something is more trustworthy than one who never does.
Provenance and consent
1. Can you trace a single record back to a consent signature? If not, you are trusting their word rather than their evidence.
2. May I see the actual consent form, in the original language? Most vendors will not produce this. It takes thirty seconds and reveals a great deal.
3. How is comprehension verified where contributors have limited literacy? A signature on an unreadable document proves nothing.
4. What is the withdrawal channel, and what happens when someone uses it? The honest answer includes what withdrawal cannot undo.
Labour and pay
5. What do you pay contributors, and will you show me the rate card? Increasingly buyers ask this, and they should. Underpaid contributors produce rushed, low-variance, high-fraud data — this is a quality question as much as an ethical one.
6. What protections apply to safety and red-team work? Look for capped exposure hours, briefing and debriefing, paid rest and unconditional opt-out.
Quality methodology
7. What proportion of items get a double-blind second pass? A number, not “we have rigorous QA”.
8. How do you measure and report inter-annotator agreement? Per batch and per label class, or it is not actionable.
9. How do you detect fraud? Duplicate fingerprinting, device and location checks, gold seeding, payment tied to QA-passed units. Fraud is constant pressure in this industry; a vendor who has not thought about it has not been operating at scale.
Commercial and contractual
10. Does the licence explicitly permit training, fine-tuning and derivative model distribution? In those words.
11. What happens when a delivery misses spec? Look for a defined acceptance window, free re-collection, and no invoice for rejected units — written into the MSA rather than promised on a call.
12. Will you tell me when I should not buy from you? The most revealing question on the list.
Signals that should worry you
Volume claims without a distribution breakdown. Certification language that blurs “aligned” with “certified”. A single headline quality number with no methodology behind it. Reluctance to discuss contributor pay. Claims to cover every language and country equally well — nobody does.
And any vendor promising they can remove data from a model you have already trained. That promise is not deliverable and offering it signals either confusion or willingness to say whatever closes the deal.
Signals that reassure
Publishing limitations alongside capabilities. Naming the cases where a competitor or a free corpus is the better answer. Reporting the numbers that reflect badly — rejected batches, quota shortfalls, low-agreement label classes. Offering a small paid pilot rather than pushing straight to a production order.
These are cheap to fake in a sales conversation and expensive to fake in a delivery report, which is why running a pilot before committing is worth its cost.
Related pages on this site
Answers
Frequently asked questions
The highest-signal questions: can you trace a record back to a consent signature; may I see the actual consent form; what do you pay contributors; what proportion of items get a double-blind second pass; and what happens contractually when a delivery misses spec. A vendor who answers all five concretely is in a small minority.
Run a paid pilot before a production order, and read the delivery report rather than the summary. Look for per-batch inter-annotator agreement, per-annotator accuracy against gold items, the adjudication log, and the list of internally rejected items. Claims are cheap; a delivery report is evidence.
For high-resource English tasks with a stable schema, a crowd platform is frequently the cheaper correct answer. For low-resource languages, strict demographic quotas, or anything needing defensible provenance, crowd platforms cannot deliver what you need — they give you whoever signs up, which is systematically the wrong distribution.
Send us your vendor questionnaire.
We will complete it and return it, usually within three business days. No procurement theatre required.
Average first response: under 6 business hours. NDAs signed same day.