Pretraining Corpora
Cleaned, deduplicated, documented text at pretraining scale, with a mixture report and per-document lineage.
- language
- domain
- unit
Training data engineering for language and generative AI
From raw sources to training-grade datasets
Datapink collects, cleans, classifies and structures multilingual text data for LLMs, machine translation, RAG and domain-specific AI — delivered in your schema with measurable quality and source-level traceability.
- Source-aware
- Multilingual
- Custom schemas
- Documented QA
01
What we build
productsFour product families. Every one of them is reachable from here in a single click.
Multilingual & Parallel Corpora
Multilingual, comparable and parallel collections — kept distinct, because they are not the same product.
- pairs
- alignment
- splits
LLM Post-training & Evaluation Data
Instruction, preference, reasoning, tool-use, safety and evaluation records, with generation and validation kept apart.
- stage
- task family
Custom Dataset Development
Your sources, your taxonomy, your schema, your delivery channel — built as a repeatable pipeline.
- schema
- refresh
02
How raw data becomes training data
how it worksSeven stages on the way in, each with an input, an action, an output and a gate that has to pass.
- 1Scope & policy
- 2Source crawl
- 3Extraction
- 4Language & domain
Multi-layer cleaning
- Input
- Normalized documents with language, script and domain labels
- Actions
- Six independent layers, A–F, applied in a fixed order
- Output
- Curated records plus a rejection report with machine-readable reason codes. Rejected records are included only where source rights and the project agreement permit.
- Quality gate
- Per-layer pass rates within the Dataset Specification
- 6Dedup & build
- 7QA & delivery
03
Multi-layer quality
qualityQuality is not one filter. Four groups of checks, each with its own defect type, decision and reason code.
Technical
Encoding, parsing and extraction validity — before anything else is scored.
LAYER_A_ENCODING
Linguistic
Boilerplate, machine filler, keyword stuffing, broken segmentation.
LAYER_C_LOWQUAL
Semantic
Duplication, contamination, off-domain and contentless documents.
DUP_NEAR_PARA
Privacy & policy
Personal data, restricted categories, policy-excluded sources.
PII_EMAIL_REDACTED
04
Built for
who it is foruse case
LLM pretraining & fine-tuning
Base corpora, domain adaptation, and the post-training data that follows.
use case
Machine translation
Parallel and comparable corpora, alignment confidence, terminology labels.
use case
Domain-specific AI
Private data handling, custom taxonomies, expert review, recurring refresh.
05
What you receive
deliveryA package, not a file — so a reviewer can reconstruct how any record got there.
- datasetJSONL / Parquet shards
- manifestsource → record lineage
- schemafields and data dictionary
- quality reportper dimension, per release
- dataset carduse, composition, limits
06
Delivered records retain an auditable lineage
quality & provenanceAgreed lineage fields, dataset versions and rejection summaries are delivered as part of the release package rather than kept only inside the pipeline.
Source-level lineage
Each delivered record carries the agreed identifiers needed to resolve it to its normalized document, retained snapshot and source record.
source_url
Versioning
Pipeline and dataset versions travel with the data, so a release can be reproduced.
pipeline_version: dp-2026.09
Rejection reasons
Pipeline removals are recorded with machine-readable reason codes; the release includes an aggregate rejection report and, where permitted, record-level rejection details.
reason_code: DUP_NEAR_PARA
Frequently asked
questions we are askedMonolingual, multilingual, comparable and parallel corpora; pretraining and continued-pretraining corpora; post-training data — instruction/SFT, preference, verifiable reasoning, tool-use and safety; grounded Q&A and RAG evaluation sets; and custom datasets built to your own schema and taxonomy.
Yes. Pretraining and continued-pretraining corpora, supervised fine-tuning (SFT) data, preference pairs for RLHF/DPO-style optimization, and evaluation sets are built as separate products, each with generation kept apart from validation and its own acceptance criteria.
Coverage is scoped per project against source availability, review capacity and an agreed quality definition, so what is feasible is confirmed before a pilot rather than assumed from a published language list.
Yes — parallel corpora (aligned translation pairs) are a distinct product from multilingual collections (the same domain across several languages, not aligned) and comparable corpora (related but not translated content). A multilingual collection is never described as parallel; the specification states which of the three you are ordering.
Delivered records resolve to their source record, retained snapshot and pipeline version through the provenance manifest. Pipeline rejections are recorded with machine-readable reason codes; the delivery includes the agreed rejection summary and any record-level details permitted by the source rights and project agreement.
Yes, where a field list or schema already exists in your stack. It is mapped from our standard fields and confirmed in the Dataset Specification before delivery.
Automated PII detection does not guarantee complete removal; high-risk fields use deterministic rules, custom recognizers, human sampling and documented residual risk. Duplicates and unsafe or policy-excluded content are handled by their own cleaning layers, each with a named method and a reason code on every rejection.
Yes. A pilot scopes a feasible language, domain or schema, agrees the quality criteria up front, and delivers a small sample release before any larger commitment.
Bring us a language, a domain or a schema.
We will help define a feasible pilot, quality criteria and a delivery format for your model.