1. Define the model or business objective
Describe what the labelled data is meant to support. Is the project for image recognition, document extraction, text classification, speech processing or another task? The objective helps determine which labels matter and which errors are most costly.
2. Document the taxonomy and edge cases
Create a label list with definitions, inclusion and exclusion rules, examples, and how to handle uncertain or overlapping cases. Explain what annotators should do when an item cannot be labelled confidently. Avoid relying on unstated assumptions.
3. Agree the output schema
Specify field names, permitted values, coordinate formats, language conventions, file structure and validation rules. If data must integrate with an existing pipeline, share a sample valid output and a sample invalid output.
4. Run a representative pilot
Select a small sample that reflects real-world variation, including difficult and unusual examples. Use the pilot to test whether the instructions are understandable, whether the schema is practical and whether edge cases are covered before scaling.
5. Define quality review
Agree how samples will be reviewed, how disagreements will be resolved, which errors require rework and how guideline changes will be communicated. Set quality targets only after agreeing the measurement method and acceptance criteria.
6. Set data handling expectations
Before transferring data, define approved access, transfer methods, retention, deletion, confidentiality and escalation procedures. Flag personal, sensitive or regulated data so appropriate controls can be considered before processing.
7. Plan delivery and handover
Confirm batch sizes, naming conventions, progress updates, issue logs, versioning, delivery dates and the person authorised to approve outputs. Keep a record of changes to the taxonomy and instructions.
Before the project begins
- Task objective and intended use documented
- Label taxonomy with examples and edge-case rules
- Output schema and validation rules approved
- Representative pilot sample selected
- Quality sampling and disagreement resolution defined
- Data access, retention and transfer requirements reviewed
- Batch plan, escalation contacts and acceptance criteria agreed
Need help putting this into practice?
DGx Creators can discuss the requirements, volumes and review process for your project. Share your brief to explore a suitable next step.
Explore AI Data Annotation ↗ Contact DGx Creators