Skip to content
Discuss your dataset

Datasets · umbrella product

Training Data for Pretraining and Post-training LLMs

Custom datasets for pretraining, domain adaptation, instruction tuning, preference optimization, verifiable reasoning, tool use, safety and model evaluation.

Design your training data pipeline

Request a pilot

01

Pretraining and post-training are different products

the two halves

They share provenance fields and almost nothing else — different units, different validators, different definitions of a correct record.

half 01

Pretraining

Continuous text at scale. The unit is a document; there is no instruction and no expected answer, so validation means extraction, language and mixture — not correctness.

half 02

Post-training

Behaviour, not vocabulary. The unit becomes a conversation, a comparison or a verifiable task, and correctness becomes checkable — which is why validation gets its own actors.

02

Pick the stage you are training

training-stage selector

Select a training stage to see the delivery composition, validation requirements and typical tasks. SFT is shown by default.

SFT

What's in the delivery

  • System, user and assistant messages, single- and multi-turn
  • Task family, language and difficulty labels on every record
  • creation_method and validation_status, recorded separately
  • Prompt and template registry, versioned

Typical tasks

  • Instruction following and task behaviour
  • Style and register control
  • Structured output that parses on the first attempt
  • Multi-turn coherence and state carrying
03

Eight stages, side by side

product matrix
Eight stages, side by side
Training stageDelivery compositionTypical tasks
PretrainingToken-ready documents, code/text mixtures, metadata, mixture reportBase / foundation model training
Continued pretrainingCurated language or domain corpusAdaptation of an existing base model
SFT / instruction tuningSystem/user/assistant messages, single- and multi-turnInstruction following, style, task behavior, structured output
Preference optimizationPrompt, candidate responses, chosen/rejected or ranking, rubricDPO and other offline preference methods
Verifiable reasoning / RFTProblems, final answers, verifier/reward signals, optional rationaleMath, code, STEM
Tool use / agenticTool definitions, calls, arguments, observations, state, final responseFunction calling, API workflows, planning, error recovery
Safety & policy behaviorBoundary prompts, allowed answers, refusals, safe alternatives, labelsSafety tuning, jailbreak resistance, reduced over-refusal
EvaluationPrivate holdouts, references, rubrics, graders, error taxonomyModel selection, regression testing, targeted data iteration
04

The taxonomy that shapes the mixture

capability & task design

Records are not sampled at random. Task family, capability, difficulty band and language are dimensions of the specification, and the mixture report states what was actually delivered.

level 01

Task family

extraction · summarisation · classification · generation · transformation · reasoning

level 02

Capability

what the model must be able to do inside that family

level 03

Difficulty band

calibrated against a reviewed sample, not asserted

level 04

Language

per record, with script — a mixture is not a monolingual set

05

Five ways a record gets made — each with its own control

production approaches

The approach is recorded on the record. It changes what validation has to prove.

approach

Human-authored

Required control: annotation guidelines, reviewer agreement, adjudication.

approach

Source-grounded transformation

Required control: evidence spans and source lineage on every record.

approach

Synthetic generation

Required control: generator, prompt version, decoding params, seed — and an independent verifier.

approach

Hybrid

Required control: which part was generated and which was written, per field.

approach

Production-log mining

Required control: consent and policy basis, plus redaction before use.

06

Generation is not validation

verification

Four separate actors. The record carries who did what, so a reviewer can disagree with any one of them.

Generation and validationThe generation column produces records — human, synthetic or hybrid — and tags each with a creation method. The validation column checks them with rule checks, a judge model and human review. The two columns meet at a shared validation_status field; neither column can set the other’s field.GenerationHuman authoredSynthetic generationHybrid, creation method taggedValidationRule checksJudge modelHuman reviewvalidation_statusNeither lane can set the other’s fieldGeneration and validationThe generation column produces records — human, synthetic or hybrid — and tags each with a creation method. The validation column checks them with rule checks, a judge model and human review. The two columns meet at a shared validation_status field; neither column can set the other’s field.GenerationHuman authoredSynthetic generationHybrid, creation method taggedvalidation_statusValidationRule checksJudge modelHuman reviewNeither lane can set the other’s field
A single LLM judge is not a sufficient quality gate.
07

What protects the evaluation

mixture, splits & decontamination

A private evaluation set is only private if no source family reaches it from the training side.

  • trainfit
  • devtune
  • testreport
  • private evalnever trained on

proportions are illustrative — the split ratio is set in the Dataset Specification

08

Three records, three shapes

sample schemas

SFT messages

{
  "messages": [
    {"role": "system", "content": "..."},
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "..."}
  ],
  "task_family": "extraction",
  "language": "eng",
  "validation_status": "accepted"
}

Illustrative — schema; no customer or production data.

tool-use record fields

09

The feedback loop

after the first release
Feedback loopA cycle: a dataset drives training, which drives evaluation, which produces an error taxonomy, which drives the next dataset version. The next dataset version feeds back into the dataset stage, shown as a dashed return path.DatasetTrainingEvaluationError taxonomyNext dataset versionThe refresh feeds back into the next production runFeedback loopA cycle: a dataset drives training, which drives evaluation, which produces an error taxonomy, which drives the next dataset version. The next dataset version feeds back into the dataset stage, shown as a dashed return path.DatasetTrainingEvaluationError taxonomyNext dataset versionThe refresh feeds back into the nextproduction run
10

LLM-specific artefacts, plus the standard package

deliverables
  • capability / task taxonomy
  • prompt & template registry
  • generator / judge / verifier registry extract
  • mixture & sequence-length report
  • annotation guidelines
  • split & contamination report
  • evaluation rubric & error taxonomy
  • sample-level lineage
  • instruction correctness
  • preference agreement
  • verifier success
  • tool success
  • groundedness
  • diversity
  • distribution balance
  • safety flags
  • leakage checks
  • format compliance
  • provenance completeness

eleven quality dimensions, each defined and measured per project

11

Related pages

related

Q&A & Instruction Data

The child product: grounded Q&A, instruction–response and RAG evaluation records.

Q&A & instruction →

Use case: LLM Pretraining & Fine-tuning

Business scenarios and how to choose between products.

Use case →

Technology

Reference architecture and the principles behind it.

Technology →
12

What we need from you to start

what you provide
  • Which training stages you need and in what order
  • Base model, tokenizer and context length, if already chosen
  • Task family and capability coverage — yours, or a node of ours
  • Volume target and the unit it is counted in
  • Schema or field list, if one already exists in your stack
  • Acceptance criteria — what must be true for a release to pass

Frequently asked

questions we are asked

Yes. Most engagements start with one stage — often SFT — and add others once the first delivery is accepted against its Dataset Specification.

Generation and validation are recorded as separate actors on every record. Judge and verifier configuration — including whether we operate them or you do — is agreed per Dataset Specification and listed in the generator/judge/verifier registry extract.

The split manifest enforces that a source family contributing to training never appears in the evaluation holdout, and the contamination report states what was checked and how.

Yes. Customer-provided prompts, templates or a task taxonomy are versioned into the prompt & template registry and run through the same validation stages as data we design.

Bring us a language, a domain or a schema.

We will help define a feasible pilot, quality criteria and a delivery format for your model.

Discuss your datasetRequest a pilot