# HonestFetch

> HonestFetch is a reading tool for AI agents. Send a URL and an optional question, and get back the smallest useful, sourced Markdown. It signs every request, obeys every site's rules, and never stores page content.

It fetches with the User-Agent `HonestFetch/0.1 (+https://honestfetch.dev/bot)`. Access during the private beta is by invitation.

## Endpoints

- [POST /v1/read](https://honestfetch.dev/v1/read): read one page. Returns Markdown with its source, or a status that says why not and a link a person can open.
- [POST /v1/preflight](https://honestfetch.dev/v1/preflight): what would happen for a URL, on what terms, without fetching the page.
- [GET /v1/usage](https://honestfetch.dev/v1/usage): what you have used this period.
- [MCP over Streamable HTTP](https://honestfetch.dev/mcp): the same two operations as the tools `read` and `preflight`, for an MCP client.
- [OpenAPI description](https://honestfetch.dev/openapi.json): machine-readable, needs no key, generated from the schemas the service validates with.

## Rules it follows, checked before every fetch

- robots.txt (the product token `HonestFetch`, then `*`), Content Signals (`ai-input=no` means no content), RSL licence terms, `Crawl-delay`, `Retry-After`, and its own exclusion list, which any site owner can join at https://honestfetch.dev/bot.
- A 401, 403, 451, challenge page or paywall is final for that request. It never retries under another identity, never uses a proxy, never solves a challenge, and never renders JavaScript.
- Page content lives only for the length of one request. No page content is stored, cached or logged.

## Limits

- 60 reads a minute per customer, at most 10 of them to any one host.
- Per destination host, for everybody at once: at most 2 requests at a time and at least a second between them, or longer when the site asks.
- An answer holds at most 1200 tokens by default, and up to 8000 on request.

## For site owners

- [POST /v1/optout](https://honestfetch.dev/v1/optout): exclude a domain and every subdomain of it, without an API key. Takes effect on the next request, before robots.txt is asked.
- [POST /v1/optout/verify](https://honestfetch.dev/v1/optout/verify): confirm the exclusion once the DNS record it names is published.

## Documents

- [About the bot](https://honestfetch.dev/bot): how to identify it, the addresses it fetches from, and how to exclude a site.
- [Privacy policy](https://honestfetch.dev/privacy)
- [Terms of service](https://honestfetch.dev/terms)
- [Data processing agreement](https://honestfetch.dev/dpa)
- [Public keys](https://honestfetch.dev/.well-known/http-message-signatures-directory): the Web Bot Auth key directory that every request signature points to.