wikis / Agentic SEO / wiki / concepts / agent-identity-web-bot-auth.md view as markdown report a mistake

Agent Identity and Web Bot Auth

Definition

Web Bot Auth is a way for automated clients to prove who they are by signing their HTTP requests, so a site can verify the sender without trusting a User-Agent string or an IP list. It builds on HTTP Message Signatures (RFC 9421, February 2024) and adds three things: a Signature-Agent header that says where the sender's public keys live, a key-directory format based on JSON Web Key Sets, and a well-known URI for that directory. It is an IETF work item, not a finished standard. It answers "who is this agent?", not "may it do this?"; permission lives in robots txt for ai and content signals and aipref.

How It Works

Cloudflare's "Forget IPs" post states the problem with the older mechanisms: User-Agent is easily spoofed and agents often send the Chrome user agent; IP ranges are shared and change; per-site bearer tokens do not scale. See ai crawlers and user agents for those methods.

The flow:

1. The agent holds a key pair and publishes the public key (as a JWK) in a key directory. Cloudflare's setup guide uses Ed25519; the draft's keyid thumbprint rules also cover RSA and EC keys.
2. Per request, it sends Signature-Input, Signature and Signature-Agent. The signature must cover @authority (target host) or @target-uri, plus the request's Signature-Agent member.
3. The verifier resolves the Signature-Agent URL, fetches keys, selects the one matching keyid, and checks the signature, created and expires.

Current IETF working-group draft (draft-ietf-webbotauth-httpsig-protocol-01, published 29 October 2026, Intended Status Standards Track, expires 2 May 2027; authors from Cloudflare and Google) requires:

• tag must be web-bot-auth; expires recommended no more than 24 hours after created.
• Signature-Agent is a Dictionary Structured Header whose members are https URIs; covered component form ("@authority" "signature-agent";key="sig1"). Earlier drafts defined it as a bare string and "deployments still send it"; signers must send the dictionary form, verifiers may accept the legacy form.
• A signed request must carry Signature-Agent. Replay protection (including nonce) is left to RFC 9421; the draft adds no nonce requirement.
• The identifier of an agent is the URL, not the key, so keys can rotate.
• Directories are JWKS served over HTTPS; at the well-known URI the media type is application/http-message-signatures-directory+json.

GET /path/to/resource HTTP/1.1
Host: origin.example.com
Signature: sig=abc==
Signature-Input: sig=("@authority" "signature-agent";key="sig");created=1700000000;expires=1700011111;keyid="ba3e64==";tag="web-bot-auth"
Signature-Agent: sig="https://signer.example.com"

(Example from the draft, line-wrapping removed; values are illustrative.)

An older companion, draft-meunier-web-bot-auth-architecture-05, is Intended Status Informational and states it "will expire on 3 September 2026". Cloudflare's docs still reference draft-meunier-http-message-signatures-directory-03 and the architecture draft revision 02, so vendor docs lag the working-group text.

Cloudflare's implementation

• Docs (updated 2026-10-08) require hosting the key directory at /.well-known/http-message-signatures-directory, signing the directory response with tag set to http-message-signatures-directory, and registering the bot via the dashboard Bot Submission Form (Verification Method: Request Signature). Cloudflare accepts Ed25519 keys.
• Verified bots (docs updated 2026-11-02): must self-identify deterministically (Web Bot Auth signature, published IP list with stable user-agent, or reverse DNS) and must obey robots.txt and crawl directives. Behaviours include Search, Agent, Training and Transact; bots are labelled Direct or Intermediary. Cloudflare says it is experimenting with forwarding end-user information via the Forwarded header (RFC 7239).
• Signed agents (post undated in the page; its Radar caption covers 21-27 August 2025, so late August 2025 is inferred): an extension of verified bots for agents directed by end users via remote browsing infrastructure. First cohort named: ChatGPT agent, Goose (Block), Browserbase, Anchor Browser. Cloudflare's own Browser Rendering signs requests. Cloudflare said enterprise customers would soon be able to act on signed agents as a group.
• Cloudflare's post on Perplexity says "ChatGPT Agent is signing http requests using the newly proposed open standard Web Bot Auth". Separately, the "Forget IPs" post quotes an OpenAI engineer that OpenAI signs all Operator requests with RFC 9421.

Cloudflare sells the bot-management products that consume these signatures; the signed-agent and verified-bot classifications are its own policy.

Key Parameters

Visa Trusted Agent Protocol (TAP)

Visa announced TAP on 17 October 2025. Its press release says it is "built upon the foundational HTTP Message Signature standard, and aligned with Web Bot Auth", citing Adobe Data Insights (August 2025) for a 4,700% rise in AI-driven visits to US retail sites (Adobe's measure of AI-driven visit share; Visa's framing). Visa's sample repository implements RFC 9421 with Ed25519 and RSA-PSS-SHA256, signature parameters including created, expires, nonce, keyId and tag, and two tags: agent-browser-auth (product details) and agent-payer-auth (checkout). TAP carries merchant-facing agent and consumer identifiers in addition to identity; see agentic commerce protocols.

When To Use

• You operate an agent or crawler that sites block by default: sign requests, publish a directory, register with the CDNs you care about.
• You run a site and want to allow specific agents past bot rules without maintaining IP lists: verify signatures at the edge, typically through a CDN that already does it.
• You sell through agents: TAP-style verification is aimed at merchants distinguishing agent checkout from fraud.
• Cloudflare's readiness scanner says not every site needs to implement it: "If your site just serves content, and doesn't make requests to other sites, you don't need it." That applies to the signing side.

Risks & Pitfalls

• Draft status. The protocol is an Internet-Draft; field names and header forms have already changed (bare string to dictionary Signature-Agent).
• Signature is not authorization. The draft says an outer signature "does not express authorization, delegation, or consent". Verified identity does not tell you whether the agent may take an action.
• Replays. A signature can be replayed within its validity window; Cloudflare recommends short expires.
• Intermediaries and transitive trust. An operator signing for many end users is trusted per operator, not per user (Cloudflare verified-bots docs).
• Coverage is partial. Many agents do not sign. Cloudflare reported that Perplexity's undeclared "stealth" crawler presented a generic Chrome-on-macOS user agent (Chrome/124.0.0.0) alongside the declared Perplexity-User/1.0; the dispute is covered in user triggered fetchers and robots. Whether specific browser agents (Atlas, Comet, Gemini auto browse) sign requests is not established in the gathered sources.
• Vendor lock of semantics. Verified-bot and signed-agent lists are Cloudflare's, with its own criteria.

Related Concepts

• ai crawlers and user agents — UA and IP verification, the older methods.
• paid access x402 and pay per crawl — pay per crawl uses signed requests to identify payers.
• agent discovery files — key directories are one of the .well-known artifacts.
• cloudflare

Sources

• raw/web_community-rfc-9421-http-message-signatures-rfc-editor.md — RFC 9421 date and role.
• raw/web_community-draft-ietf-webbotauth-httpsig-protocol-01.md — draft -01 dates, status, authors.
• raw/github_doc-draft-ietf-webbotauth-httpsig-protocol-md.md — draft text: Signature-Agent, tag, directory, replay.
• raw/web_community-draft-meunier-web-bot-auth-architecture-05.md — Informational architecture draft and expiry.
• raw/web_community-forget-ips-using-cryptography-to-verify-bot-and-agent-traffi.md — problem statement, OpenAI quote, example headers.
• raw/web_community-the-age-of-agents-cryptographically-recognizing-agent-traffi.md — signed agents and first cohort.
• raw/web_community-web-bot-auth-cloudflare-bot-solutions-docs.md — setup requirements and header table.
• raw/web_community-verified-bots-cloudflare-bot-solutions-docs-2.md — verified-bot criteria and labels.
• raw/web_community-perplexity-is-using-stealth-undeclared-crawlers-to-evade-web.md — ChatGPT Agent signing claim; Chrome UA example.
• raw/github_doc-readme-md-13.md — Cloudflare reference implementations.
• raw/web_community-visa-introduces-trusted-agent-protocol-an-ecosystem-led-fram.md — TAP announcement, Web Bot Auth alignment.
• raw/github_doc-tap-agent-readme-md.md — TAP signature algorithms and tags.
• raw/github_doc-readme-md-14.md — TAP merchant-facing goals.
• raw/web_community-introducing-the-agent-readiness-score-check-to-see-if-your-s.md — "you don't need it" guidance.