A low-resource language pair
There is not enough bitext to train on, and what exists is concentrated in one register. You need source text in both languages before alignment is even worth attempting.
Use case · MT teams
Training data for machine translation
Monolingual data to build on, parallel and comparable corpora to train on, alignment and filtering you can inspect, and terminology labels that survive into the release.
01
Three ways MT data goes wrong
your situationEach one points at a different product. The decision helper below turns the symptom into a starting point.
Domain-specific terminology
General-domain output is fluent and wrong. Terms drift between documents, and there is no labelled terminology to hold the model to.
Noisy existing bitext
You have pairs, but they carry misalignment, copied targets, untranslated segments and duplicates — and no confidence score to filter on.
02
Four things, in the order they usually happen
what we build for thisMonolingual data
Cleaned source text in each language, so a low-resource side has something to mine and back-translate from.
- per language
- domain-scoped
Parallel & comparable corpora
Aligned translation pairs, and topically matched texts where true parallel data does not exist. The two are never mixed or relabelled.
- aligned pairs
- comparable
Alignment & filtering
Document matching, segment alignment, length-ratio, punctuation, numeric and entity consistency, untranslated and copied-target detection.
- alignment_score
- rule filters
Terminology & domain metadata
Domain labels and terminology lists carried on the pair, so evaluation can be sliced by the thing you actually care about.
- domain
- terminology
03
Start from the symptom, not the catalogue
decision helper| If this is your situation | Start here | Why |
|---|---|---|
| Not enough bitext in the pair | Monolingual corpora | Build the source side first; mine and back-translate from it |
| Bitext exists but is noisy | Multilingual & parallel corpora | Alignment confidence plus rule filters, with rejection reasons reported and record-level rejects included where permitted |
| Fluent but wrong in your domain | Custom dataset development | Domain corpus plus terminology labels and expert review |
| No way to measure a change | Evaluation holdouts | Leakage-aware splits and a private test set |
| Topically related texts only | Comparable corpora | Cross-lingual retrieval and mining, explicitly not labelled parallel |
Most MT projects use two of these at once. The pilot decides the split.
04
What lands in your training directory
formatsFormat follows your toolchain, not ours. Schema detail lives on the product page, not here.
- TMX
- TSV / CSV
- Marian-compatible plain text
- JSONL
- Parquet
- your schema
05
What we measure on an MT project
quality controls that matter hereAlignment quality
Agreement between the assigned alignment and a reviewed sample of pairs.
how measured · alignment score + human sample
Language accuracy per side
Language and script correctness scored independently for source and target.
how measured · cascaded LID, per side
Terminology consistency
Agreement of a term's rendering across the corpus, against your term list.
how measured · term list match, per domain
Leakage-aware splits
No source family appears in both training data and the evaluation holdout.
how measured · split manifest validation
Bring us a language, a domain or a schema.
We will help define a feasible pilot, quality criteria and a delivery format for your model.