Skip to content
Discuss your dataset

Process

How raw data becomes training data

Ten stages, each with inputs, actions, outputs and a quality gate — configured per Dataset Specification.

Discuss your dataset

See Quality & Provenance

Pretraining corpus

the stage list below is shown for the selected example — the stages are the same, the actions and gates differ

  1. artifact · Dataset Specification v1 with acceptance criteria

  2. Requirements & corpus specification

    Input
    Your use case, target model stage, languages, domains and constraints
    Actions
    Scope the unit, the schema, the thresholds and what acceptance means
    Output
    Dataset Specification v1, signed off before anything is fetched
    Quality gate
    Acceptance criteria are measurable and agreed in writing
  3. 1Source discovery & data mining

    Source registry, allowlist/denylist, sampling report

  4. artifact · Crawl manifest, raw snapshot, fetch log, coverage stats

  6. 3Extraction & structural parsing

    Normalized documents, parser statistics, extraction warnings

  7. 4Language & script identification

    Language, script and confidence per document and segment

  8. 5Domain & content classification

    Taxonomy labels, genre, review bands

  9. artifact · Curated records and a rejection report for layers A–F; rejected records are included where source rights and the project agreement permit

  11. 7Corpus / dataset construction

    Dedup, alignment, splits, mixture, versioning, model registry

  12. 8Quality assurance

    Automated validation, stratified human sampling, full review where required

  13. 9Documentation & delivery

    Deliverables package, dataset card, checksums, changelog

A

What each stage leaves behind

sample artifacts

Three of the artefacts produced along the way.

stage 2 · Crawl manifest

{
  "source_id": "src_1042",
  "http_status": 200,
  "content_hash": "sha256:9f2c…",
  "robots_decision": "allowed"
}

Illustrative — schema; no customer or production data.

B

From first brief to a versioned release

project lifecycle
From first brief to a versioned release
StepWhat comes out of it
DiscoveryBrief, source policy, target schema, acceptance metrics
Feasibility sampleSmall real sample plus a risk and coverage report
Paid pilotFull pipeline run on limited volume
Production runScaling the approved configuration
AcceptanceQA package and customer validation
RefreshVersioned delta or periodic rebuild
C

Pilot, production, refresh

after the first release
Pilot, production, refreshThree nodes in a loop: a paid pilot produces a feasibility report, a production run produces a v1 release and quality report, and a refresh produces a delta and changelog. The refresh feeds back into the next production run, shown as a dashed return path.PilotfeasibilityreportProductionv1 release +quality reportRefreshdelta vN +changelogRefresh feeds the next production runPilot, production, refreshThree nodes in a loop: a paid pilot produces a feasibility report, a production run produces a v1 release and quality report, and a refresh produces a delta and changelog. The refresh feeds back into the next production run, shown as a dashed return path.PilotfeasibilityreportProductionv1 release +quality reportRefreshdelta vN +changelogRefresh feeds the next production run

Bring us a language, a domain or a schema.

We will help define a feasible pilot, quality criteria and a delivery format for your model.

Discuss your datasetRequest a pilot