HonestFetch

HonestFetch is a reading tool for AI agents. It fetches one page at a time, on behalf of a named customer, and returns a compact summary of the text. It is probably in your logs because a developer asked it to read a page of yours.

How to identify it

User-Agent: HonestFetch/0.1 (+https://honestfetch.dev/bot)
robots.txt product token: HonestFetch
Request signature: Every request carries Signature, Signature-Input and Signature-Agent headers (Web Bot Auth, RFC 9421). The agent header is sig1="https://honestfetch.dev";type=directory.
Public keys: /.well-known/http-message-signatures-directory
Operator: HallinCreations, Sweden. Write to bot@honestfetch.dev.
Declared content use: immediate, meaning the text answers one question now and is not stored, indexed, or reused.

If those headers are missing, it was not us. Anybody can put a string in a User-Agent. Only we can sign with the keys in that directory, so the signature is the thing to check. Send us the log line and we will confirm it.

Addresses it fetches from

2.29.16.92
2a01:4f9:c015:683e::1

How to control it

HonestFetch obeys all of these, and checks them before it fetches a page.

robots.txt. A Disallow for HonestFetch, or for *, is obeyed. We look for our own token first, then the wildcard, as RFC 9309 requires.
Content Signals. ai-input=no means we return no content from your pages.
RSL. If you publish a licence we cannot meet, we refuse the read and tell the agent what your terms are.
Crawl-delay. Honoured, and it only ever slows us down. Without one we still wait at least a second between requests to your site and never open more than two at once.
429 and 503. Retry-After is honoured exactly, for every customer at once, not just the one who triggered it.

Exclude your site entirely

This takes effect on the very next request, before we ask your server anything at all, including for robots.txt. It covers every subdomain: excluding example.com also excludes www.example.com and docs.example.com.

We verify that you control the domain, because an exclusion anybody could file against anybody would be a way to attack your competitors using our name.

What we keep from this form, and for how long, is in the privacy policy.

The answer names a TXT record to publish at _honestfetch-optout.yourdomain and the value to put in it. Publish it, then confirm:

curl -X POST https://honestfetch.dev/v1/optout/verify \
  -H 'Content-Type: application/json' \
  -d '{"domain":"example.com"}'

Prefer the command line for both steps? The first one is the same shape:

curl -X POST https://honestfetch.dev/v1/optout \
  -H 'Content-Type: application/json' \
  -d '{"domain":"example.com","contact":"you@example.com","method":"dns"}'

We keep the domain, which of the three methods proved it, a hash of your address, and two timestamps. Not the address itself, and no free text at all.

What it never does

It never pretends to be a browser, and never sends a User-Agent but its own.
It never rotates addresses or uses proxies to get around a block.
It never solves a CAPTCHA or any other challenge.
It never logs in, and never passes a paywall or a metering wall.
It never renders JavaScript, and never submits a form or takes any other action on your site.
It never stores your pages. The text lives for the length of one request and is then gone. There is no cache, no archive, and no column in any table that could hold a page.
A 401, 403, 451, challenge page or paywall is final for that request. We do not retry it under another identity.

Contact

Write to bot@honestfetch.dev. If you are reporting traffic you did not expect, a single log line with the request headers is the most useful thing you can send: it lets us tell our requests from somebody using our name.

We answer complaints by stopping the traffic first and discussing it afterwards.