Datasets · custom build
Custom dataset development
Your sources, your taxonomy, your schema, your delivery channel — built as a repeatable pipeline rather than a one-off export.
Fourteen decisions that are yours to make
what can be customisedEach one is written into the Dataset Specification and applied identically on every release.
- Customer-provided data transformation
- Source allowlist / denylist
- Custom taxonomy & labels
- Custom JSON/JSONL/CSV/TSV/XML/TMX/Parquet schema
- Document / chunk / pair / conversation unit
- Custom ID rules
- Tokenization & segmentation requirements
- Naming, sharding, compression, partitioning
- Custom rejection codes
- Customer evaluation rules
- Enrichment: entities, terminology, metadata, citations
- Cloud / storage delivery
- Recurring delta releases
- Refresh & versioning
Schema review comes before anything is built
how a custom build startsA brief becomes a schema before it becomes a pipeline. That is the step that prevents a second release.
- 1Brief
Schema review
- Input
- Your field list, or the export you already receive from somewhere else
- Actions
- Map every field, name what is missing, agree the unit and the ID rules
- Output
- A target schema both sides have signed off
- Quality gate
- Every field has a source, a type and an acceptance rule
- 3Feasibility sample
- 4Paid pilot
- 5Production
Our field, your field
schema mapping| Our field | Your field |
|---|---|
| dp_doc_id | record_key |
| language | lang_iso3 |
| script | lang_script |
| domain | topic_l1 |
| subdomain | topic_l2 |
| source_url | origin_uri |
| fetched_at | ingest_ts |
| quality_score | qc_score |
| pipeline_version | build_tag |
Illustrative — mapping; no customer or production data. Fields with no counterpart on either side are the useful part of a schema review — they are the ones that turn into questions.
Final fields, validation rules and delivery channels are fixed in the Dataset Specification before production.
Frequently asked
questions we are askedYes — it enters at the schema review with its own history. We map what exists, state what cannot be reconstructed (lineage is the usual gap), and agree what the first release under our pipeline will and will not claim.
Where your policy requires it, yes — the pipeline can run against your storage and compute; the schema review is where that boundary is agreed.
Yes. Custom rejection codes are mapped into the Dataset Specification and carried on the rejection summary of every release.
Every release is versioned against the signed-off schema, with a changelog stating what changed between deltas.
A schema change is scoped as a new schema review; the current specification stays in force for releases already in flight.
Bring us a language, a domain or a schema.
We will help define a feasible pilot, quality criteria and a delivery format for your model.