Skip to content
Discuss your dataset

Technology

Reference architecture for corpus engineering

Capabilities and principles, not a dependency list — tool versions live in dated technical notes.

Request a technical architecture review

See the process

01

Six stages, four storage layers, one lineage spine

architecture
Which stage reads and writes which storage layerA matrix of four storage layers — raw, normalized, curated and delivered — against six pipeline stages: acquisition, extraction, language & domain, cleaning, dedup & construction, and QA & documentation. Acquisition writes raw storage; extraction reads raw and writes normalized storage; language & domain reads normalized storage; cleaning reads normalized storage and writes curated storage; dedup & construction writes curated storage; QA & documentation reads curated storage and writes delivered storage. Raw storage is written once and never rewritten by any later stage.AcquisitionExtractionLanguage & domainCleaningDedup & constructionQA & documentationRaw storageimmutableNormalized storageCurated storageDelivered storage●writes·○reads·immutableWhich stage reads and writes which storage layerA matrix of four storage layers — raw, normalized, curated and delivered — against six pipeline stages: acquisition, extraction, language & domain, cleaning, dedup & construction, and QA & documentation. Acquisition writes raw storage; extraction reads raw and writes normalized storage; language & domain reads normalized storage; cleaning reads normalized storage and writes curated storage; dedup & construction writes curated storage; QA & documentation reads curated storage and writes delivered storage. Raw storage is written once and never rewritten by any later stage.Acquisition●Raw storage — writesimmutableExtraction○Raw storage — readsimmutable●Normalized storage — writesLanguage & domain○Normalized storage — readsCleaning○Normalized storage — reads●Curated storage — writesDedup & construction●Curated storage — writesQA & documentation○Curated storage — reads●Delivered storage — writes●writes○readsimmutable
02

Eight things the architecture commits to

principles

HTTP-first crawling

Static fetch is the default; browser rendering is a policy-approved exception, and the lane used is recorded on the document.

Format-aware extraction

HTML, PDF and office documents, with reading order, tables, formulas and captions treated as content rather than noise.

Scalable curation

Deduplication, multilingual intelligence and layered cleaning that behave the same at pilot and production scale.

Benchmark before adoption

A model enters the registry only after it is benchmarked on project-relevant data — never because it is new.

Orchestration & lineage

Run events in an OpenLineage-style shape, so a record's history is queryable rather than reconstructed.

Storage layers

Immutable raw, normalized, curated and delivery — each stage reads one and writes the next, and raw is never rewritten.

Human in the loop

Sampling, adjudication and domain review are part of the architecture, not a manual step bolted on at the end.

Machine-readable transparency

Croissant-format metadata and a training-content transparency pack travel with the release.

03

What is recorded before a model is used

model registry
What is recorded before a model is used
FieldWhat it records
namethe model, as published
version / commitexact, not a range
licenseas stated by the publisher
checksumof the artefact we actually loaded
eval datewhen it was benchmarked here
languageswhat it was benchmarked on
benchmarkthe project-relevant set, not a leaderboard
approved usewhich stages may use it

Bring us a language, a domain or a schema.

We will help define a feasible pilot, quality criteria and a delivery format for your model.

Discuss your datasetRequest a pilot