Skip to content
Discuss your dataset

Services · curation

Multi-layer text cleaning and curation

Quality comes from a sequence of independent filters — each with a defect type, a decision, a metric and a machine-readable reason code.

Discuss your dataset

Explore our process

01

Six filters, not one cleaning step

what this stage does

A single “cleaning” box hides which defect was removed and why. Splitting it into independent layers means every decision is attributable — and every threshold is negotiable.

01

Independent filters

The layers run in a fixed order. A document rejected at layer B is never scored at layer F, so cost and evidence both stay traceable.

02

A decision per record

Keep and reject are both written to the record, with the layer, the metric and a machine-readable reason code.

03

Configured, not fixed

Thresholds come from the Dataset Specification — a pretraining corpus and a preference dataset do not share a recipe.

04

Auditable output

Rejection summary included; record-level rejected data is delivered only where source rights and the project agreement permit. Removals are aggregated by reason code, so they can be reviewed rather than trusted.

02

The six layers

the central visual

A and E are shown expanded; each band opens into defect type, a keep/reject pair, who decided, the metric and the reason code.

  1. Defect type
    Truncated encoding, unparsable markup, empty extraction
    Example
    keep: utf-8 decodes cleanly, 1 840 characters extracted reject: ��� <div id="wrap">
    Detection
    automated
    Metric
    extraction validity
    Reason code
    LAYER_A_ENCODING

  2. Defect type
    Navigation, cookie banners, share widgets, footers, pagination
    Example
    keep: Body text remaining after the boilerplate strip reject: “Accept all cookies” · “Share on” · site footer
    Detection
    automated
    Metric
    boilerplate ratio
    Reason code
    LAYER_B_BOILERPLATE

  3. Defect type
    Machine-generated filler, keyword stuffing, broken segmentation
    Example
    keep: Coherent paragraphs with consistent sentence segmentation reject: “buy buy buy cheap cheap best price best price now now”
    Detection
    automated
    Metric
    linguistic quality score
    Reason code
    LAYER_C_LOWQUAL

  4. Defect type
    Exact, near and substring duplicates; evaluation leakage
    Example
    keep: Cluster representative, kept by the ordered pipeline reject: Near-duplicate of a kept record, or an evaluation-set overlap
    Detection
    automated
    Metric
    near-duplicate cluster membership
    Reason code
    DUP_NEAR_PARA

  5. Defect type
    Personal data, restricted categories, policy-excluded sources
    Example
    keep: Contact: [EMAIL_REDACTED] reject: Contact: [email protected], +49 30 …
    Detection
    automated + human
    Metric
    residual-risk sampling
    Reason code
    PII_EMAIL_REDACTED

  6. Defect type
    Off-domain, contentless, template-only documents
    Example
    keep: On-domain document within the Dataset Specification’s topic scope reject: Template placeholder page with no domain-relevant content
    Detection
    automated
    Metric
    domain relevance sampling
    Reason code
    LAYER_F_OFFDOMAIN
03

Deduplication is an ordered pipeline

layer D · duplication

Order matters as much as the thresholds: a cheaper filter never runs after a more expensive one, and clustering happens once.

Deduplication runs as an ordered ladderFive steps run in a fixed order, each descending a step and removing a defined category of duplicate: Content hash removes exact duplicates, reason code DUP_EXACT; Normalized exact match removes exact duplicates after normalization, reason code DUP_NORM; MinHash / LSH near-duplicate clustering removes near-duplicates, reason code DUP_NEAR_PARA; Substring overlap removes contained passages, reason code DUP_SUBSTR; Semantic (optional), shown dashed, removes semantic near-duplicates, reason code DUP_SEM. Each cluster keeps a cluster identifier, a representative record and the reason the rest were dropped.Content hashexact duplicatesDUP_EXACTNormalized exact matchexact after normalizationDUP_NORMMinHash / LSH near-duplicatenear-duplicatesDUP_NEAR_PARASubstring overlapcontained passagesDUP_SUBSTRSemantic (optional)semantic (optional)DUP_SEMEach cluster keeps a cluster identifier, a representative record and the reason the rest were dropped.Deduplication runs as an ordered ladderFive steps run in a fixed order, each descending a step and removing a defined category of duplicate: Content hash removes exact duplicates, reason code DUP_EXACT; Normalized exact match removes exact duplicates after normalization, reason code DUP_NORM; MinHash / LSH near-duplicate clustering removes near-duplicates, reason code DUP_NEAR_PARA; Substring overlap removes contained passages, reason code DUP_SUBSTR; Semantic (optional), shown dashed, removes semantic near-duplicates, reason code DUP_SEM. Each cluster keeps a cluster identifier, a representative record and the reason the rest were dropped.Content hashexact duplicatesDUP_EXACTNormalized exact matchexact after normalizationDUP_NORMMinHash / LSH near-duplicatenear-duplicatesDUP_NEAR_PARASubstring overlapcontained passagesDUP_SUBSTRSemantic (optional)semantic (optional)DUP_SEMEach cluster keeps a cluster identifier,a representative record and the reasonthe rest were dropped.
04

What automated detection does not promise

layer E · safety, privacy & policy
E.1

Deterministic rules

Structured identifiers — emails, phone numbers, account and document numbers — are matched by rule, not by model confidence.

E.2

Custom recognizers

Project-specific categories are added per Dataset Specification and versioned with the pipeline.

E.3

Human sampling

A stratified sample of redactions is reviewed, and the residual risk is stated in the dataset card rather than implied away.

05

Cleaning is configuration, not a universal recipe

thresholds differ by product

The same six layers, three different settings. This is why the page shows a method and not a number.

Cleaning thresholds by product
LayerPretraining corpusParallel corpusPost-training dataset
Layer B boilerplateAggressive — volume tolerates lossModerate — alignment needs contextStrict — prompts must read cleanly
Layer C linguisticThreshold on document levelPer side, both languages scoredPer turn, plus style compliance
Layer D duplicationNear-duplicate clustering at scalePair-level, plus copied-target detectionExact + leakage-aware split check
Layer E privacyRedaction, retain structureRedaction applied to both sides consistentlyRedaction plus boundary-case review
Layer F usefulnessDomain relevance samplingNot applied — alignment carries relevanceTask-family balance instead
Human reviewStatistical sampleSample of low-confidence pairsAll records of a flagged class

Values are configuration patterns, not published thresholds. The agreed settings live in the Dataset Specification and are restated in the dataset card of every release.

06

Rejection report preview

what comes back with the data

Removals are a deliverable, not a side effect. The report aggregates by reason code so a reviewer can challenge any layer.

rejection report · aggregated by reason code

  • LAYER_B_BOILERPLATE
  • DUP_NEAR_PARA
  • LAYER_C_LOWQUAL
  • LAYER_F_OFFDOMAIN
  • PII_EMAIL_REDACTED
  • LAYER_A_ENCODING

Bar lengths are shape only. Real shares are measured per release against the Dataset Specification and shipped inside the quality report — no rejection figure is published as a site-wide claim.

JSONL

{
  "id": "dp_doc_000318",
  "layer": "D",
  "reason_code": "DUP_NEAR_PARA"
}

Illustrative — schema; no customer or production data.

07

Inputs and outputs

interfaces

Inputs

  • Normalized documents with language, script and domain labels
  • Dataset Specification with per-layer thresholds and acceptance criteria
  • Source policy decisions from the source registry
  • Customer taxonomy or schema, where one exists

Outputs

  • Curated records carrying every layer decision
  • Rejection summary included; record-level rejected data delivered only where source rights and the project agreement permit
  • Cluster manifest with representatives and drop reasons
  • Quality report entry per layer, per release

Frequently asked

questions we are asked

Yes. Customer-provided collections enter at this stage with their own source policy. We run the same six layers, report what each one removed, and hand back a cleaned release plus the rejection summary — record-level rejected data only where source rights and the project agreement permit; the original is never overwritten.

Thresholds are set per project in the Dataset Specification, based on the product type and the acceptance criteria you define — a pretraining corpus and a preference dataset are configured differently, as shown above.

Rejected documents are removed from the delivered dataset. Every rejection is written to the record with its layer and reason code, and the aggregated rejection summary ships alongside the data; record-level rejected data is delivered only where source rights and the project agreement permit, so removals can be reviewed.

Yes. Project-specific reason codes are defined per Dataset Specification and versioned with the pipeline, the same way custom recognizers are added at layer E.

No. Automated PII detection reduces risk but does not guarantee complete removal; high-risk fields combine deterministic rules, custom recognizers and human sampling, with the residual risk documented in the dataset card.

Bring us a language, a domain or a schema.

We will help define a feasible pilot, quality criteria and a delivery format for your model.

Discuss your datasetRequest a pilot