Skip to content
Discuss your dataset

Services · intelligence

Language, script and domain intelligence

Cascaded language identification, script validation and hybrid domain classification — calibrated per project with explicit review bands.

Discuss your dataset

See the full process

01

The same document is scored three times

three levels

A source can be reliably one language while individual paragraphs are not. Scoring at one level only is how code-switched and boilerplate-heavy documents slip through.

level 01

Source

What the registry says this domain publishes, and in which languages — a prior, not a verdict.

level 02

Document

The assigned language, script and confidence that travels on the record.

level 03

Paragraph / segment

Where code-switching, quotations and untranslated blocks are actually caught.

02

Four signals, one decision, three bands

decision layer
Language identification bandsFour signals — a multilingual identification model, script validation, a baseline detector and project rules — feed one decision layer, which sorts each document into one of three labelled bands, each with its own shape mark: a confident band (teal, check mark), an uncertain review band (amber, question mark) and a rejected band (red, cross).LID modelScript checkBaseline detectorProject rulesDecision layerConfident band✓Review band?Rejected band✕
03

Where a document is filed

subject taxonomy

Technology → Software → Cloud/DevOps

Medicine → Clinical → Cardiology

Legal → Contracts → Employment

Finance → Markets → Equities

  • news
  • statute
  • forum thread
  • product documentation
  • academic paper
  • transcript
  • review
  • reference entry

genre and document-type chips, applied alongside the subject node

04

Your labels, versioned

custom taxonomies

A customer taxonomy is imported, versioned and applied exactly as given. Where it disagrees with ours, yours wins and the mapping is recorded.

custom

Imported as given

Node names, hierarchy and definitions are kept — we do not silently normalise them to ours.

custom

Versioned

taxonomy_version travels on every record, so a relabelled release is distinguishable.

custom

Mapped, not merged

Where both taxonomies apply, the crosswalk is a separate artefact.

05

What lands in the uncertain band, and why

review bands

The band is not a failure state. It is where short texts, mixed-language documents and low-resource languages go to be looked at by a person.

band 01

Short or fragmentary text

Too little signal for a confident call; the score is honest about it rather than rounded up.

band 02

Code-switching & quotation

Documents that genuinely contain two languages — a single label would be wrong either way.

band 03

Low-resource languages

Where the model has least support, review capacity is budgeted in advance.

06

How the classifier is held to account

validation

Metrics are measured per project on an approved labelled sample, never quoted as a house figure.

Macro / micro F1

Per-class and overall agreement with the approved labelled sample.

how measured · measured per project

Precision / recall

Reported per critical class, not averaged into one number.

how measured · measured per project

Low-confidence share

Proportion of documents that land in the review band.

how measured · measured per release

Confusion on critical classes

Where the classifier confuses classes that matter to you.

how measured · confusion matrix, per project

Confusion matrix · illustrative shape
true ↓ / pred →ukrrusbulsrp
ukr■■■■■■··
rus■■■■■■··
bul·■■■■■■·
srp··■■■■■

Cell density is shape only. Real counts come from the approved sample and ship with the quality report.

07

What the record carries afterwards

sample metadata

JSON

{
  "id": "dp_doc_000412",
  "language": "ukr",
  "script": "Cyrl",
  "language_score": 0.99,
  "domain": "legal",
  "subdomain": "legal.contracts.employment",
  "genre": "statute",
  "audience_level": "professional",
  "review_band": "confident",
  "taxonomy_version": "dp-tax-2026.07"
}

Illustrative — field names; no customer or production data.

The record's language chip: ukr · Cyrl · 0.99

Frequently asked

questions we are asked

The identification stack covers a broad set, but the number that matters is the one we can hold to an agreed quality definition on your sources — which is settled per project against a labelled sample, not published as a count.

They are routed for human review rather than assigned a low-confidence label automatically. A reviewer resolves the language, script or domain call, and the decision is written to the record.

Yes. Your taxonomy is imported and versioned as given; where it disagrees with ours, yours wins and the mapping between the two is recorded as a separate artefact.

Code-switching is caught at the paragraph or segment level, where a single document-level label would be wrong either way, and routed to the review band rather than forced to one language.

It is scoped per project against the Dataset Specification — some corpora need only document-level labels, others need the paragraph or segment level where code-switching matters most.

Bring us a language, a domain or a schema.

We will help define a feasible pilot, quality criteria and a delivery format for your model.

Discuss your datasetRequest a pilot