Web Bot Auth for Verifying AI Crawlers

A server log that says GPTBot, Googlebot, or another published name proves only that the client sent that string. IP validation and reverse DNS can improve confidence when a provider publishes stable infrastructure, but proxies, intermediaries, and agentic requests complicate the model. Web Bot Auth applies HTTP Message Signatures so a verifier can test possession of a registered private key.

Verification answers a bounded question: whether the request signature validates under the registered key and supported protocol. It does not certify the truth of every payload, authorize the requested action, or prove the ultimate human or model purpose. Authentication belongs before authorization, not instead of it.

Cloudflare’s implementation expects a Signature-Agent structured string pointing to an HTTPS key directory, Signature-Input describing the signed components and parameters, and Signature carrying the cryptographic value. The signature-agent component must itself be included in the covered component list. Cloudflare currently accepts registered Ed25519 keys from the directory.

Choose stable components such as @authority and the signature-agent field, then add method, path, or other components according to the authorization risk. Unsupported components or serialization parameters cause verification failure. Implementation code must target the verifier’s documented draft rather than assuming every HTTP Message Signatures example is interchangeable.

A captured signed request may be replayed if the verifier accepts the same signature later. Cloudflare recommends short expiration windows and notes that nonce validation is not currently backed by a seen-nonce database. Treat created, expires, authority, method, and target components as part of the threat model, not optional decoration.

A minute-long expiry may be appropriate for ordinary crawl requests, but clock skew and queueing need measured tolerance. Do not sign mutable intermediary-controlled headers unless the delivery path preserves them exactly. When body integrity matters, a content digest can help only if intermediaries will not transform the representation.

Register the key directory and bot metadata, test signed requests against the documented validation endpoint, and log the exact verification result. Cloudflare describes 200 for known valid keys, 401 for correctly formatted but unknown or failed identities, and 400 for malformed requests in its test flow. Keep test and production keys separate.

At the origin or edge, create an allow decision only after verification and only for the paths, methods, and rates the crawler needs. Do not bypass authentication, administrative routes, write endpoints, or rate controls. Preserve an emergency revocation path for compromised keys and rotate keys without a gap in the published directory.

Record verifier version, registered agent, key identifier, signature time, expiry, covered-component set, route, decision, and failure class. Do not log private keys, raw authorization data, or unnecessary signed bodies. Aggregate invalid signatures separately from unknown agents and expired requests so an implementation bug is not mislabeled as hostile traffic.

Web Bot Auth is a meaningful improvement over name matching, especially as signed agents become common. It is still an evolving ecosystem built from standards and implementation profiles. Maintain protocol fixtures, rotate keys, follow verifier updates, and keep fallbacks explicit. A secure system knows exactly what evidence caused an allow decision.