Skip to content

Quality & provenance

What “training-grade” means

Measurable quality, source-level lineage and documented limitations — defined per Dataset Specification, not asserted site-wide.

01

Three things have to be true at once

definition
pillar 01

Validity

The data is what it claims to be: it decodes, parses, is in the language it says, and belongs to the domain it is filed under.

pillar 02

Provenance

Every record resolves back to a source, a snapshot and the pipeline version that produced it.

pillar 03

Reproducibility

The same retained source snapshots, Dataset Specification, pipeline and model versions, configuration and fixed random seeds reproduce the release; differences between releases are recorded in the changelog.

02

Where a record comes from

lineage
A delivered record resolves to its sourcesA delivered record has five fields, each resolving to a source, joined on a shared spine: source_id resolves to the source registry, raw_snapshot resolves to the retained snapshot, normalized_doc resolves to the normalized document, curated_record resolves to the layer decisions, and dataset_version resolves to the provenance manifest.delivered recordsource_idresolves to the source registryraw_snapshotresolves to the retained snapshotnormalized_docresolves to the normalized documentcurated_recordresolves to the layer decisionsdataset_versionresolves to the provenance manifest●●●●●A delivered record resolves to its sourcesA delivered record has five fields, each resolving to a source, joined on a shared spine: source_id resolves to the source registry, raw_snapshot resolves to the retained snapshot, normalized_doc resolves to the normalized document, curated_record resolves to the layer decisions, and dataset_version resolves to the provenance manifest.delivered recordsource_idresolves to the source registryraw_snapshotresolves to the retained snapshotnormalized_docresolves to the normalized documentcurated_recordresolves to the layer decisionsdataset_versionresolves to the provenance manifest●●●●●
03

Eleven dimensions, measured separately

quality dimensions

Each card states what is measured and how. None of them is reported as a single percentage that stands for the dataset as a whole.

Validity

Documents that decode, parse and extract to non-empty text.

how measured · automated validation, full pass

Language accuracy

Agreement between assigned language and a reviewed labelled sample.

how measured · cascaded LID + human sample

Domain relevance

Documents a reviewer accepts for the stated taxonomy node.

how measured · stratified human sample

Extraction quality

Reading order, tables, captions and formulas preserved as specified.

how measured · parser checks + sample

Duplication rate

Records removed or clustered as exact, near or substring duplicates.

how measured · ordered dedup pipeline

Alignment quality

Agreement of assigned alignment with a reviewed sample of pairs.

how measured · alignment score + sample

Groundedness & correctness

Answers supported by their cited evidence span.

how measured · rule checks + human review

Post-training correctness

Instruction correctness, preference agreement, verifier and tool success.

how measured · per-record validators

Diversity & distribution

Coverage across task family, difficulty, language and source.

how measured · mixture report

Safety & privacy flags

Records carrying a redaction, exclusion or policy flag.

how measured · rules + custom recognizers

Provenance completeness

Records with a full, resolvable lineage chain.

how measured · manifest validation, full pass

04

Four levels of review

how much is checked by a person

The level is chosen per class of object and stated in the dataset card.

01

Automated validation

Applied to every object. Deterministic, reproducible, no opinion.

02

Human review of a statistical sample

Stratified sample sized to the claim being made.

03

Human review of all objects of a class

Applied where a class carries elevated risk.

04

Full human annotation

Every object seen by a person, with adjudication.

05

An ordered pipeline, not a switch

deduplication
Deduplication runs as an ordered ladderFive steps run in a fixed order, each descending a step and removing a defined category of duplicate: Content hash removes exact duplicates, reason code DUP_EXACT; Normalized exact match removes exact duplicates after normalization, reason code DUP_NORM; MinHash / LSH near-duplicate clustering removes near-duplicates, reason code DUP_NEAR_PARA; Substring overlap removes contained passages, reason code DUP_SUBSTR; Semantic (optional), shown dashed, removes semantic near-duplicates, reason code DUP_SEM. Each cluster keeps a cluster identifier, a representative record and the reason the rest were dropped.Content hashexact duplicatesDUP_EXACTNormalized exact matchexact after normalizationDUP_NORMMinHash / LSH near-duplicatenear-duplicatesDUP_NEAR_PARASubstring overlapcontained passagesDUP_SUBSTRSemantic (optional)semantic (optional)DUP_SEMEach cluster keeps a cluster identifier, a representative record and the reason the rest were dropped.Deduplication runs as an ordered ladderFive steps run in a fixed order, each descending a step and removing a defined category of duplicate: Content hash removes exact duplicates, reason code DUP_EXACT; Normalized exact match removes exact duplicates after normalization, reason code DUP_NORM; MinHash / LSH near-duplicate clustering removes near-duplicates, reason code DUP_NEAR_PARA; Substring overlap removes contained passages, reason code DUP_SUBSTR; Semantic (optional), shown dashed, removes semantic near-duplicates, reason code DUP_SEM. Each cluster keeps a cluster identifier, a representative record and the reason the rest were dropped.Content hashexact duplicatesDUP_EXACTNormalized exact matchexact after normalizationDUP_NORMMinHash / LSH near-duplicatenear-duplicatesDUP_NEAR_PARASubstring overlapcontained passagesDUP_SUBSTRSemantic (optional)semantic (optional)DUP_SEMEach cluster keeps a cluster identifier,a representative record and the reasonthe rest were dropped.
06

What we do, and what we do not claim

privacy & safety

Deterministic rules

Structured identifiers matched by rule, not by model confidence.

Custom recognizers

Project-specific categories, versioned with the pipeline.

Documented residual risk

Stated in the dataset card, with the sampling method behind it.

07

Decisions are recorded, not inferred

source, license & policy review

robots.txt is an access-management protocol, not a licence; content-use rules are assessed separately. Rights-reservation signals are recorded as policy inputs, not interpreted as legal conclusions.

Source, license & policy review — example decisions
sourcepolicy signaldecisionreviewer_rolereviewed_at
src_1042robots allowed; no rights-reservation signal foundincludesource-policy reviewer2026-08-14
src_0551terms restrict automated reuseexcludesource-policy reviewer2026-08-14
src_0207rights signal present; scope unclearreviewlegal/policy review2026-08-15
src_0318licensed under project agreementincludesource-policy reviewer2026-08-15

Illustrative — decision records; no customer or production data.

08

Dataset card, Croissant metadata, registry extract

documentation
Dataset card · preview
FieldValue
intended usepretraining, continued pretraining
compositiondocuments, one language, taxonomy node
collection methodfocused crawl under approved source policy
cleaninglayers A–F, thresholds per specification
known limitationsstated, with residual risk
pipeline_versiondp-2026.09
not suitable foruse outside the documented languages, domains and intended training stage without additional validation

Croissant 1.0 · registry extract

{
  "@context": {
    "@language": "en",
    "@vocab": "https://schema.org/",
    "citeAs": "cr:citeAs",
    "column": "cr:column",
    "conformsTo": "dct:conformsTo",
    "cr": "http://mlcommons.org/croissant/",
    "data": {"@id": "cr:data", "@type": "@json"},
    "dataType": {"@id": "cr:dataType", "@type": "@vocab"},
    "dct": "http://purl.org/dc/terms/",
    "extract": "cr:extract",
    "field": "cr:field",
    "fileProperty": "cr:fileProperty",
    "fileObject": "cr:fileObject",
    "fileSet": "cr:fileSet",
    "format": "cr:format",
    "includes": "cr:includes",
    "jsonPath": "cr:jsonPath",
    "key": "cr:key",
    "md5": "cr:md5",
    "parentField": "cr:parentField",
    "path": "cr:path",
    "recordSet": "cr:recordSet",
    "references": "cr:references",
    "regex": "cr:regex",
    "repeated": "cr:repeated",
    "replace": "cr:replace",
    "sc": "https://schema.org/",
    "separator": "cr:separator",
    "source": "cr:source",
    "subField": "cr:subField",
    "transform": "cr:transform"
  },
  "@type": "sc:Dataset",
  "name": "dp_monolingual_illustrative",
  "description": "Illustrative Croissant metadata example. Field names and structure only, generated for the documentation of the quality and provenance page; not a published dataset and not a volume, count or throughput claim. The license and distribution values below are placeholders, not a real licence grant or a real file.",
  "conformsTo": "http://mlcommons.org/croissant/1.0",
  "license": "https://example.com/dataset-licence",
  "url": "https://datapink.net/quality-provenance/",
  "version": "2026.9.0",
  "datePublished": "2026-09-19",
  "creator": {
    "@type": "sc:Organization",
    "name": "Datapink",
    "url": "https://datapink.net/"
  },
  "distribution": [
    {
      "@type": "cr:FileObject",
      "@id": "source-archive",
      "name": "source-archive",
      "description": "Illustrative — delivery archive placeholder; no customer or production data. The file does not exist, and the sha256 below is a fabricated example checksum, not a real digest.",
      "contentUrl": "https://example.com/deliveries/corpus-example.tar",
      "encodingFormat": "application/x-tar",
      "sha256": "0000000000000000000000000000000000000000000000000000000000000000"
    },
    {
      "@type": "cr:FileSet",
      "@id": "documents",
      "name": "documents",
      "description": "Illustrative — document shard set; no customer or production data.",
      "containedIn": {"@id": "source-archive"},
      "encodingFormat": "application/jsonlines",
      "includes": "*.jsonl"
    }
  ],
  "recordSet": [
    {
      "@type": "cr:RecordSet",
      "@id": "documents-record-set",
      "name": "documents",
      "description": "One row per accepted document, illustrative field names only.",
      "field": [
        {
          "@type": "cr:Field",
          "@id": "documents-record-set/source_id",
          "name": "source_id",
          "description": "Resolvable identifier for the source the document was extracted from.",
          "dataType": "sc:Text",
          "source": {
            "fileSet": {"@id": "documents"},
            "extract": {"jsonPath": "$.source_id"}
          }
        },
        {
          "@type": "cr:Field",
          "@id": "documents-record-set/text",
          "name": "text",
          "description": "Extracted document text.",
          "dataType": "sc:Text",
          "source": {
            "fileSet": {"@id": "documents"},
            "extract": {"jsonPath": "$.text"}
          }
        }
      ]
    }
  ],
  "citeAs": "Datapink. Illustrative Croissant metadata example. Generated for documentation purposes; not a citation for a published dataset."
}

Illustrative — field names; no customer or production data.

09

How a takedown reaches the data

data removal
  1. step 01

    Request

    URL or source, what is asked for, the basis, and how to reach the requester.

  2. step 02

    Verification

    The requester's relationship to the material is confirmed before anything moves.

  3. step 03

    Assessment

    Assessed against the source policy and applicable obligations; the decision is recorded.

  4. step 04

    Dataset-version impact

    Affected records are traced through lineage and carried into the next version.

Data Rights & Removal

10

Channels are agreed, not advertised

security & delivery
  • Delivery channels are agreed per project and named in the contract, not listed on this page
  • Checksums accompany every shard; manifests are signed off at acceptance
  • Project isolation and NDA terms are settled before any sample is exchanged
  • No customer data appears in any sample, artefact or diagram on this site

Bring us a language, a domain or a schema.

We will help define a feasible pilot, quality criteria and a delivery format for your model.