Skip to content
Discuss your dataset

Datasets · corpora

Multilingual, comparable and parallel corpora

Three products that are routinely confused with each other. We keep them apart, because they are built differently, measured differently and used for different things.

Request a pilot
01

Three products, not three words for one

terminology

This distinction is the single most common source of a mis-specified MT project.

product 01

Multilingual

Independent corpora in several languages. No relationship between the documents is claimed or checked.

product 02

Comparable

Topically matched texts across languages. Useful for cross-lingual retrieval, terminology and mining — not for supervised MT.

product 03

Parallel

Aligned translation pairs with a confidence score. This is the only one of the three that trains a translation model directly.

Multilingual, comparable and parallel compared
Row labelMultilingualComparableParallel
Relationship between sidesnone claimedtopicaltranslation
Unitdocumentdocument pairaligned segment pair
Typical usemultilingual pretraining, coveragecross-lingual retrieval, mining, terminologyMT training, alignment, quality estimation
Key metriclanguage accuracymatch qualityalignment quality
What goes wrong if mislabelledcoverage overstatedtreated as training pairs—
02

Retrieval, alignment, filtering, estimation, review

how a parallel corpus is built

Alignment is a measured step with a score attached, not an assertion. Nothing in this pipeline is described as “AI alignment” — it is retrieval, rules and quality estimation.

  1. 1Document retrieval & matching
  2. 2Sentence / segment alignment
  3. 3Rule filters

  5. Quality estimation

    Input
    Aligned candidate pairs that survived the rule filters
    Actions
    Score each pair; route low-confidence pairs to human sample
    Output
    alignment_score on every pair, plus the rejected set
    Quality gate
    Sampled agreement between score and human judgement
  6. 5Human sample
03

Twelve things the pipeline actually does

capabilities
capability

Document matching

Candidate document pairs from comparable collections.

capability

Segment alignment

Sentence and segment level, with an explicit confidence.

capability

Many-to-one / one-to-many

Alignment policies chosen per project, not assumed 1:1.

capability

Source / target validation

Language and script checked independently on each side.

capability

Length-ratio filters

Pairs that cannot be translations of each other, removed by rule.

capability

Punctuation & numeric consistency

Numbers, dates and entities that disagree across the pair.

capability

Alignment confidence

A score on the pair, not a binary accept.

capability

Untranslated / copied target

Targets that repeat the source, caught explicitly.

capability

Boilerplate & duplicate pairs

Pair-level dedup, after document-level dedup.

capability

Terminology & domain labels

Carried on the pair so evaluation can be sliced.

capability

Leakage-aware splits

One source family never spans train and evaluation.

capability

Comparable, kept separate

Topically matched text is labelled comparable — never parallel.

04

What lands in your training directory

formats

TSV / CSV

Pair records across TSV/CSV, TMX and aligned plain text
Row labelTSV / CSV (shown)TMXAligned plain text
Pair perrowtranslation unitline index across two files
Metadataextra columnsproperties on the unitsidecar manifest
Best forfiltering and inspectiontranslation-memory toolingMarian-style training pipelines
05

One pair record

sample schema

TSV / CSV

{
  "pair_id": "dp_pair_000001",
  "source_language": "deu",
  "target_language": "eng",
  "source_text": "...",
  "target_text": "...",
  "domain": "legal.contracts",
  "alignment_score": 0.97,
  "document_id": "dp_parallel_doc_0124",
  "source_id": "src_0551",
  "split": "train"
}

Illustrative — schema; no customer or production data.

06

Measured per side, and on the pair

quality checks

Alignment quality

Agreement between assigned alignment and a reviewed sample of pairs.

how measured · alignment score + human sample

Language accuracy per side

Language and script scored independently for source and target.

how measured · cascaded LID, per side

Duplication rate

Pair-level duplicates after document-level dedup.

how measured · ordered dedup, pair level

Terminology consistency

Agreement of a term's rendering across the corpus.

how measured · term list match, per domain

07

To scope a parallel project

what you provide
  • The language pair, and which side is the source of truth
  • Existing bitext, if you have it — including the noisy parts
  • Domain, and a term list if one exists
  • Whether comparable data is acceptable where parallel does not exist
  • Your evaluation holdout situation
  • Target format and the toolchain it feeds

Frequently asked

questions we are asked

Sometimes — through comparable-corpus mining and alignment, with the confidence scores exposed so you can decide what to keep. Where mining cannot reach a usable quality, we say so at the feasibility sample rather than delivering low-confidence pairs as if they were parallel.

Only where the project scope allows it, and it is recorded as the creation method on every affected pair — never presented as human-produced bitext.

The alignment policy — 1:1, many-to-one or one-to-many — is agreed per project and applied consistently; it is never assumed to be 1:1 by default.

Yes. Alignment score, domain and terminology labels are carried on every pair so you can apply your own thresholds after delivery.

Splits are drawn at source-family level, the same rule used across every corpus type, so one source never spans train and evaluation.

Bring us a language, a domain or a schema.

We will help define a feasible pilot, quality criteria and a delivery format for your model.

Discuss your datasetRequest a pilot