Datasets · corpora

Monolingual corpora

Cleaned, deduplicated, documented text collections in one language — for pretraining, continued pretraining, language adaptation, embeddings, classification, RAG and research.

Request a pilot

Discuss your dataset

What this corpus is for

intended uses

Each use changes the specification — the unit, the cleaning thresholds and what counts as acceptance.

pretraining
continued pretraining (CPT)
language adaptation
embeddings
classification
RAG corpora
linguistic research
terminology extraction

Eight decisions that define your corpus

configurable dimensions

These are settled in the Dataset Specification before any crawling starts, and restated in the dataset card of every release.

Language & script

One language per corpus, script validated per document.

Domain / genre

Your taxonomy or ours, versioned and applied at document level.

Time range & freshness

Publication window, and whether the corpus is rebuilt or extended.

Unit

Document, paragraph or segment — decided before extraction, not after.

Size

Documents, words, characters or tokens, against a named tokenizer.

Cleaning & dedup rules

Per-layer thresholds and the dedup order that applies.

PII / safety policy

Which categories are redacted, which are excluded at source.

Metadata & schema

Field list, ID rules, sharding, compression, partitioning.

Six stages, each with a quality gate

how it is built

The product pipeline is the general process narrowed to this corpus type. Stage 5 is expanded.

1Requirements & corpus specification
2Source discovery & policy
3Crawling, extraction & normalization
4Language & script identification
Multi-layer cleaning (A–F)
Input
Normalized documents with language, script and domain labels
Actions
Layers A–F applied in order; every decision written to the record
Output
Curated records plus a rejection report with reason codes; rejected records are included only where permitted
Quality gate
Per-layer pass rates within the Dataset Specification thresholds
6Dedup, construction & documentation

See the full ten-stage process

Measurable parameters, not a quality score

quality checks

There is no single number that means “good corpus”. Each dimension is defined, measured by a named method, and reported per release.

Validity

Share of documents that decode, parse and extract to non-empty text.

how measured · automated validation, full pass

Language accuracy

Agreement between the assigned language and a reviewed labelled sample.

how measured · cascaded LID + human sample

Domain relevance

Share of documents a domain reviewer accepts for the stated taxonomy node.

how measured · stratified human sample

Duplication rate

Records removed or clustered as exact, near or substring duplicates.

how measured · ordered dedup pipeline, cluster level

Provenance completeness

Share of records carrying a full, resolvable lineage chain.

how measured · manifest validation, full pass

Supported formats

delivery

Supported formats compared
Row label	JSONL	Parquet	Plain text
Unit per row	one document	one document	one document per file
Metadata	inline fields	typed columns	sidecar manifest
Sharding	by size, gzip	by row group	by directory
Best for	streaming, mixed pipelines	columnar analytics, large scale	legacy tooling

Supported formats compared
Row label	JSONL	Parquet	Plain text
Unit per row	one document	one document	one document per file
Metadata	inline fields	typed columns	sidecar manifest
Sharding	by size, gzip	by row group	by directory
Best for	streaming, mixed pipelines	columnar analytics, large scale	legacy tooling

Supported formats compared
Row label	JSONL	Parquet	Plain text
Unit per row	one document	one document	one document per file
Metadata	inline fields	typed columns	sidecar manifest
Sharding	by size, gzip	by row group	by directory
Best for	streaming, mixed pipelines	columnar analytics, large scale	legacy tooling

One record, every field accounted for

sample schema

Field names, ID rules and the tokenizer the size is measured against are all part of the specification.

{
  "id": "dp_doc_000001",
  "text": "...",
  "language": "ukr",
  "script": "Cyrl",
  "language_score": 0.99,
  "domain": "technology.software",
  "source_id": "src_1042",
  "source_url": "https://example.org/article",
  "published_at": "2026-01-15",
  "fetched_at": "2026-02-03T12:45:00Z",
  "quality_score": 0.93,
  "license_or_policy": "reviewed_status_code",
  "pipeline_version": "dp-2026.09"
}

id: string
text: string
language: string
script: string
language_score: double
domain: string
source_id: string
source_url: string
published_at: string
fetched_at: string
quality_score: double
license_or_policy: string
pipeline_version: string

dp_doc_000001.txt
  ...document text...

dp_doc_000001.manifest.json
{
  "id": "dp_doc_000001",
  "language": "ukr",
  "script": "Cyrl",
  "source_id": "src_1042",
  "pipeline_version": "dp-2026.09"
}

// your field list, mapped from ours
{
  "your_id_field": "id",
  "your_text_field": "text"
}

Illustrative — schema; no customer or production data.

Illustrative — schema; no customer or production data. license_or_policy records the reviewed status of the source — it is a policy input, not a legal conclusion.

A package, not a file

what you receive

Every release ships the same bundle, so an auditor can reconstruct how each record got there.

dataset filesJSONL / Parquet shards
schema & data dictionaryfield-level definitions
dataset cardintended use, composition, limitations
Croissant metadatamachine-readable, JSON-LD
provenance manifestsource_id → record lineage
quality reportper dimension, per release
rejection summaryaggregated by reason code
checksumssha256 per shard
split manifesttrain / dev / test / private eval
changelogversioned deltas

What we need from you to start

what you provide

Target language and script, and whether variants count as one corpus
Domain or taxonomy — yours, or a node of ours
Source situation: your own data, public discovery, or licensed sources
Volume target and the unit it is counted in
Schema or field list, if one already exists in your stack
Acceptance criteria — what must be true for a release to pass

Frequently asked

questions we are asked

Yes — coverage is scoped per project against source availability, review capacity and an agreed quality definition. We state what is feasible before a pilot rather than publishing a language count we would have to defend for every pair.

Volume targets are agreed per Dataset Specification against the unit you need — documents, words, characters or tokens against a named tokenizer — rather than published as a general figure.

Yes. Customer-provided collections enter the pipeline under their own source policy and run through the same language, cleaning and quality stages as discovered sources.

A refresh extends an existing corpus with new material under the same specification; a rebuild reruns the full pipeline against a revised specification. Both are versioned with a changelog.

Yes, where a field list or schema already exists in your stack. It is mapped from our standard fields and confirmed per Dataset Specification before delivery.

Bring us a language, a domain or a schema.

We will help define a feasible pilot, quality criteria and a delivery format for your model.

Discuss your dataset Request a pilot