Validity
The data is what it claims to be: it decodes, parses, is in the language it says, and belongs to the domain it is filed under.
Quality & provenance
Measurable quality, source-level lineage and documented limitations — defined per Dataset Specification, not asserted site-wide.
The data is what it claims to be: it decodes, parses, is in the language it says, and belongs to the domain it is filed under.
Every record resolves back to a source, a snapshot and the pipeline version that produced it.
The same retained source snapshots, Dataset Specification, pipeline and model versions, configuration and fixed random seeds reproduce the release; differences between releases are recorded in the changelog.
Each card states what is measured and how. None of them is reported as a single percentage that stands for the dataset as a whole.
Documents that decode, parse and extract to non-empty text.
how measured · automated validation, full pass
Agreement between assigned language and a reviewed labelled sample.
how measured · cascaded LID + human sample
Documents a reviewer accepts for the stated taxonomy node.
how measured · stratified human sample
Reading order, tables, captions and formulas preserved as specified.
how measured · parser checks + sample
Records removed or clustered as exact, near or substring duplicates.
how measured · ordered dedup pipeline
Agreement of assigned alignment with a reviewed sample of pairs.
how measured · alignment score + sample
Answers supported by their cited evidence span.
how measured · rule checks + human review
Instruction correctness, preference agreement, verifier and tool success.
how measured · per-record validators
Coverage across task family, difficulty, language and source.
how measured · mixture report
Records carrying a redaction, exclusion or policy flag.
how measured · rules + custom recognizers
Records with a full, resolvable lineage chain.
how measured · manifest validation, full pass
The level is chosen per class of object and stated in the dataset card.
Applied to every object. Deterministic, reproducible, no opinion.
Stratified sample sized to the claim being made.
Applied where a class carries elevated risk.
Every object seen by a person, with adjudication.
Structured identifiers matched by rule, not by model confidence.
Project-specific categories, versioned with the pipeline.
Stated in the dataset card, with the sampling method behind it.
robots.txt is an access-management protocol, not a licence; content-use rules are assessed separately. Rights-reservation signals are recorded as policy inputs, not interpreted as legal conclusions.
| source | policy signal | decision | reviewer_role | reviewed_at |
|---|---|---|---|---|
| src_1042 | robots allowed; no rights-reservation signal found | include | source-policy reviewer | 2026-08-14 |
| src_0551 | terms restrict automated reuse | exclude | source-policy reviewer | 2026-08-14 |
| src_0207 | rights signal present; scope unclear | review | legal/policy review | 2026-08-15 |
| src_0318 | licensed under project agreement | include | source-policy reviewer | 2026-08-15 |
Illustrative — decision records; no customer or production data.
| Field | Value |
|---|---|
| intended use | pretraining, continued pretraining |
| composition | documents, one language, taxonomy node |
| collection method | focused crawl under approved source policy |
| cleaning | layers A–F, thresholds per specification |
| known limitations | stated, with residual risk |
| pipeline_version | dp-2026.09 |
| not suitable for | use outside the documented languages, domains and intended training stage without additional validation |
Illustrative — field names; no customer or production data.
URL or source, what is asked for, the basis, and how to reach the requester.
The requester's relationship to the material is confirmed before anything moves.
Assessed against the source policy and applicable obligations; the decision is recorded.
Affected records are traced through lineage and carried into the next version.
We will help define a feasible pilot, quality criteria and a delivery format for your model.