Web-Bot-Auth-Checker

Bots that prove who they are

A User-Agent string can say anything, and an IP range changes whenever a crawler moves to another cloud. Web Bot Auth gives automated clients — search crawlers, AI assistants browsing on a user's behalf, monitoring services — a verifiable identity instead: each request is signed with HTTP Message Signatures (RFC 9421) using a key the operator publishes. The protocol is an IETF draft (draft-meunier-webbotauth-httpsig-protocol) already deployed by Cloudflare's verified-bots programme and by agents such as OpenAI's ChatGPT agent.

A signed request carries three fields:

• Signature-Agent — a Structured Fields dictionary naming where the signer's keys live, for instance sig1="https://chatgpt.com";
• Signature-Input — the covered components (at least @authority and the Signature-Agent member), created, expires, keyid (the JWK thumbprint of the signing key, RFC 7638) and tag="web-bot-auth";
• Signature — the signature bytes.

The key directory

The origin named in Signature-Agent publishes its keys at /.well-known/http-message-signatures-directory (directory draft): a JWK Set served as application/http-message-signatures-directory+json, in which each key's kid is its thumbprint. The directory response is itself signed with each key it lists — a signature tagged http-message-signatures-directory over @authority;req and Content-Digest — which proves that whoever serves the directory holds the private keys and cannot list somebody else's key. The member can also point at a JWK Set (type=jwks_uri) or an OAuth client metadata document (type=cimd).

What the checker verifies

1. The profile. A web-bot-auth signature must have created, expires and a thumbprint keyid, cover @authority (or @target-uri), not be expired or created in the future, and should not live longer than 24 hours.
2. Signature-Agent. The checker parses the dictionary (and the older single-string form, with a warning), finds which member the signature covers, and works out the URL to resolve. A member that is not https, not a bare origin for a directory, or not covered by the signature is reported.
3. The directory. Status 200 without redirects, the right media type, a JWK Set of usable public keys, kid equal to the thumbprint, no private members, nbf and exp respected — and the directory's own possession-proof signatures with their Content-Digest.
4. The request signature with the directory key whose thumbprint is the keyid, through the same RFC 9421 signature base the HTTP Message Signatures verifier builds. The result says what the request can be attributed to: the directory URL when a directory vouches for the key, or only the key itself when it came from Keys you already trust.

The examples are the test vectors of the protocol draft's Appendix E, signed with the RFC 9421 test keys: an Ed25519 request with the signed directory of E.2.3, an RSA-PSS request with an unsigned directory, and the legacy string form of Signature-Agent. The signature mechanism underneath is explained in HTTP Message Signatures (RFC 9421) for APIs and bots.