Skip to content
Discuss your dataset

Services · acquisition

Web data mining and controlled crawling

Source discovery, focused crawling and file ingestion under an approved source policy — with provenance captured at URL level.

Discuss your sources

Quality & Provenance

01

Nothing is crawled before it is on the register

source discovery

Seed URLs, sitemaps and feeds are scored by topic, language, quality and risk. Each source gets a decision — include, exclude or review — with a reviewer and a date attached.

Source registry sample
source_iddomaintypelanguagedecisionreviewed_at
src_1042example.orgnews archiveukr
  • include
2026-08-14
src_0551example.delegal registerdeu
  • exclude
2026-08-14
src_0207example.netforumeng
  • review
2026-08-15
src_0318example.iolicensed archivemixed
  • include
2026-08-15
src_0774example.frinstitutionalfra
  • include
2026-08-16

Illustrative — source registry; no customer or production data.

02

Budgets, limits and canonical URLs

focused crawling
control 01

Crawl budgets

Per source and per release, so a single domain cannot dominate the corpus by accident.

control 02

Rate limits & backoff

Politeness settings per host, with retries that back off rather than hammer.

control 03

Canonical URLs

Normalization and redirect tracking, so the same document is not fetched under three identities.

control 04

Change detection

content_hash comparison decides whether a re-fetch produces a new snapshot at all.

03

HTTP first, browser only when policy approves it

how pages are fetched
Acquisition lanesTwo lanes read a shared source registry and write to a shared raw snapshot. The default lane fetches static HTML over HTTP, drawn solid and blue. A second lane, used only where the source policy approves it, renders the page in a browser, drawn dashed and amber. Both lanes record which lane was used.Source registryHTTP fetch (default)Browser render (by exception)Raw snapshotDefault laneOnly where source policy approves itRecords which lane was used
04

A crawl you can inspect afterwards

observability
  • Coverage statistics per source and per release
  • Error statistics split by class — 4xx, 5xx, timeout, parse
  • Fetch log with one line per attempt, not per success
  • Skipped-URL reasons, so exclusions are visible
  • Crawl manifest attached to the raw snapshot
  • Configuration version recorded on every fetch
  • Delta runs diffed against the previous manifest
  • Nothing is re-fetched silently
05

Nine fields captured on every document

provenance
  • source_url — the canonical URL, after redirect resolution
  • fetch_time — when this snapshot was taken
  • http_status — what the server answered
  • mime_type — what came back, not what was expected
  • content_hash — sha256 of the bytes, for change detection
  • crawler_config_version — which crawl configuration produced it
  • user_agent — how we identified ourselves
  • robots_decision — the access decision that was applied
  • rights_signal_version — which rights-reservation vocabulary was read
06

A refresh is a delta, not a rebuild

incremental updates

Re-crawls compare against the previous manifest: unchanged documents keep their identity, changed ones get a new snapshot, and removals are recorded rather than silently dropped.

  • previous manifest
  • → re-fetch under budget
  • → content_hash diff
  • → new / changed / gone
  • → delta release
07

What robots.txt is, and what it is not

responsible acquisition
08

Inputs and outputs

interfaces

Inputs

  • Seed URLs, sitemaps, feeds or an archive to ingest
  • Language, domain and risk criteria for source scoring
  • Source policy — yours, or drafted together
  • Crawl budget and refresh cadence

Outputs

  • Source registry with include / exclude / review decisions
  • Immutable raw snapshot and crawl manifest
  • Fetch log plus coverage and error statistics
  • Provenance fields on every document
09

Crawl manifest

sample artifact

JSON

{
  "crawl_id": "dp_crawl_2026_09_0004",
  "source_id": "src_1042",
  "seed_url": "https://example.org/archive",
  "fetched": "[count] · illustrative",
  "skipped": "[count] · illustrative",
  "errors": {"4xx": "[count]", "5xx": "[count]", "timeout": "[count]"},
  "rate_limit": "[AGREED PER SOURCE]",
  "crawler_config_version": "dp-crawl-2026.08",
  "robots_decision": "allowed",
  "rights_signal_version": "tdmrep-2026.02"
}

Illustrative — field names; no customer or production data.

Frequently asked

questions we are asked

No. Authentication, paywalls, CAPTCHAs and anti-bot measures are access controls, and we do not work around them. If a source behind one is important to your corpus, it has to arrive through a licence or through your own access, recorded in the source policy.

Sources are scored against the language, domain, quality and risk criteria agreed for the project, then decided include, exclude or review on the source registry before any fetch runs.

The registry decision is re-reviewed against the new policy. A source that no longer qualifies is moved to exclude and any further crawling under the old decision stops.

Yes. A customer-provided archive enters at ingestion with its own source policy, recorded on the registry the same way a crawled source would be.

Refresh cadence is agreed per project and run as a delta against the previous manifest, so a refresh does not require rebuilding the corpus from scratch.

Bring us a language, a domain or a schema.

We will help define a feasible pilot, quality criteria and a delivery format for your model.

Discuss your datasetRequest a pilot