Crawler Policy

1. What our crawler does

Source discovery, focused crawling and file ingestion under an approved source policy, with provenance captured at URL level.

We collect material from the public web to build text corpora and training datasets. Every source is assessed and entered on a source registry, with a decision to include, exclude or review it, before anything is fetched from it.

Crawling is focused rather than exhaustive. It runs against an approved source policy for the project, under crawl budgets per source and per release, and it fetches the documents that policy covers rather than everything a domain holds.

Pages are fetched over HTTP first. Browser rendering is used only where the source policy approves it for a specific source, not as the default way we read the web.

Provenance is captured for each document at URL level: the canonical URL, when it was fetched, what the server answered, what came back, a hash of the bytes, which crawler configuration produced it, the user-agent we sent, the access decision that was applied, and which rights-reservation vocabulary was read. That record is what lets a source be questioned, and a document be traced, after the fact.

Requests from our crawler are ordinary requests for content a source already serves publicly.

2. How we identify ourselves

The user-agent string our crawler sends, and how an operator can recognize and manage our requests.

Our crawler identifies itself with a distinct user-agent string, published here so an operator can recognize and manage our requests: DatapinkBot

The same user-agent is used for every fetch, including where a page has to be rendered in a browser, and it is recorded against every document we keep, so a fetch in your logs can be matched to a document in our record.

An operator can use that string to allow, restrict or exclude our requests through the same controls used for any other crawler, and to tell us apart from traffic that merely claims the name. If you see the string used in a way that does not match this policy, tell us — it is not us.

We do not present our crawler as ordinary visitor traffic, and we do not rotate identities to avoid being recognized.

3. robots.txt and rights-reservation signals

How robots.txt and other rights-reservation signals are read: as an access-management protocol and policy inputs, not as a licence or a legal conclusion.

robots.txt is an access-management protocol, not a licence; content-use rules are assessed separately. Rights-reservation signals are recorded as policy inputs, not interpreted as legal conclusions.

We read robots.txt before fetching from a source and apply what it says to our own user-agent. The decision that was applied is recorded against each fetch, so it can be checked afterwards.

Machine-readable rights-reservation signals are collected and recorded with the version of the vocabulary they were read under. They feed the assessment of a source; they are not treated as settling the question of what may be done with the content.

What a source permits in terms of use is assessed separately, as part of the source policy, and a source can be excluded on that ground even where access is technically open.

4. Rate limits and access controls

Crawling is rate-limited and stays within what a source makes openly accessible.

We do not bypass access controls, CAPTCHAs, paywalls, authentication or anti-bot measures.

Requests are rate-limited per host, with retries that back off rather than repeat quickly, and with crawl budgets that cap how much is taken from any one source.

A source that would require working around a login, a paywall, a CAPTCHA or an anti-bot measure is excluded at the registry rather than worked around. If such a source matters to a project, it has to arrive through a licence or through the customer's own access, recorded in the source policy.

If our requests are still causing a problem for your site, tell us and we will reduce or stop them. A rate that suits your infrastructure can be agreed and applied per source.

5. How to request removal

How a source operator or rights holder can ask us to stop crawling a source or to remove material already collected.

To request that we stop crawling a source, or to ask about material already in a dataset, use the Data Rights & Removal form.

Tell us the URL or source identifier, what you are asking for, and on what basis you are asking. The request is logged, your relationship to the material is verified, it is assessed against the source policy, and affected records are traced through lineage into the next dataset version.

You can also write to our Legal Team at [email protected]. Excluding our user-agent in robots.txt stops further crawling on its own, but it does not reach material already collected — for that, send the request.