---
title: "Agent Identity and Web Bot Auth"
type: concept
tags: [web-bot-auth, http-message-signatures, rfc-9421, signed-agents, verified-bots, visa-tap, ietf]
created: 2026-11-04
updated: 2026-11-04
confidence: medium
sources: ["raw/web_community-rfc-9421-http-message-signatures-rfc-editor.md", "raw/web_community-draft-ietf-webbotauth-httpsig-protocol-01.md", "raw/github_doc-draft-ietf-webbotauth-httpsig-protocol-md.md", "raw/web_community-draft-meunier-web-bot-auth-architecture-05.md", "raw/web_community-forget-ips-using-cryptography-to-verify-bot-and-agent-traffi.md", "raw/web_community-the-age-of-agents-cryptographically-recognizing-agent-traffi.md", "raw/web_community-web-bot-auth-cloudflare-bot-solutions-docs.md", "raw/web_community-verified-bots-cloudflare-bot-solutions-docs-2.md", "raw/web_community-perplexity-is-using-stealth-undeclared-crawlers-to-evade-web.md", "raw/github_doc-readme-md-13.md", "raw/web_community-visa-introduces-trusted-agent-protocol-an-ecosystem-led-fram.md", "raw/github_doc-tap-agent-readme-md.md", "raw/github_doc-readme-md-14.md", "raw/web_community-introducing-the-agent-readiness-score-check-to-see-if-your-s.md"]
---

# Agent Identity and Web Bot Auth

## Definition

Web Bot Auth is a way for automated clients to prove who they are by signing their HTTP requests, so a site can verify the sender without trusting a `User-Agent` string or an IP list. It builds on HTTP Message Signatures (RFC 9421, February 2024) and adds three things: a `Signature-Agent` header that says where the sender's public keys live, a key-directory format based on JSON Web Key Sets, and a well-known URI for that directory. It is an IETF work item, not a finished standard. It answers "who is this agent?", not "may it do this?"; permission lives in [[concepts/robots-txt-for-ai]] and [[concepts/content-signals-and-aipref]].

## How It Works

Cloudflare's "Forget IPs" post states the problem with the older mechanisms: `User-Agent` is easily spoofed and agents often send the Chrome user agent; IP ranges are shared and change; per-site bearer tokens do not scale. See [[concepts/ai-crawlers-and-user-agents]] for those methods.

The flow:

1. The agent holds a key pair and publishes the public key (as a JWK) in a key directory. Cloudflare's setup guide uses Ed25519; the draft's `keyid` thumbprint rules also cover RSA and EC keys.
2. Per request, it sends `Signature-Input`, `Signature` and `Signature-Agent`. The signature must cover `@authority` (target host) or `@target-uri`, plus the request's `Signature-Agent` member.
3. The verifier resolves the `Signature-Agent` URL, fetches keys, selects the one matching `keyid`, and checks the signature, `created` and `expires`.

Current IETF working-group draft (`draft-ietf-webbotauth-httpsig-protocol-01`, published 29 October 2026, Intended Status Standards Track, expires 2 May 2027; authors from Cloudflare and Google) requires:

- `tag` must be `web-bot-auth`; `expires` recommended no more than 24 hours after `created`.
- `Signature-Agent` is a Dictionary Structured Header whose members are `https` URIs; covered component form `("@authority" "signature-agent";key="sig1")`. Earlier drafts defined it as a bare string and "deployments still send it"; signers must send the dictionary form, verifiers may accept the legacy form.
- A signed request must carry `Signature-Agent`. Replay protection (including `nonce`) is left to RFC 9421; the draft adds no nonce requirement.
- The identifier of an agent is the URL, not the key, so keys can rotate.
- Directories are JWKS served over HTTPS; at the well-known URI the media type is `application/http-message-signatures-directory+json`.

```http
GET /path/to/resource HTTP/1.1
Host: origin.example.com
Signature: sig=abc==
Signature-Input: sig=("@authority" "signature-agent";key="sig");created=1700000000;expires=1700011111;keyid="ba3e64==";tag="web-bot-auth"
Signature-Agent: sig="https://signer.example.com"
```

(Example from the draft, line-wrapping removed; values are illustrative.)

An older companion, `draft-meunier-web-bot-auth-architecture-05`, is Intended Status Informational and states it "will expire on 3 September 2026". Cloudflare's docs still reference `draft-meunier-http-message-signatures-directory-03` and the architecture draft revision 02, so vendor docs lag the working-group text.

### Cloudflare's implementation

- Docs (updated 2026-10-08) require hosting the key directory at `/.well-known/http-message-signatures-directory`, signing the directory response with `tag` set to `http-message-signatures-directory`, and registering the bot via the dashboard Bot Submission Form (Verification Method: Request Signature). Cloudflare accepts Ed25519 keys.
- **Verified bots** (docs updated 2026-11-02): must self-identify deterministically (Web Bot Auth signature, published IP list with stable user-agent, or reverse DNS) and must obey `robots.txt` and crawl directives. Behaviours include Search, Agent, Training and Transact; bots are labelled Direct or Intermediary. Cloudflare says it is experimenting with forwarding end-user information via the `Forwarded` header (RFC 7239).
- **Signed agents** (post undated in the page; its Radar caption covers 21-27 August 2025, so late August 2025 is inferred): an extension of verified bots for agents directed by end users via remote browsing infrastructure. First cohort named: ChatGPT agent, Goose (Block), Browserbase, Anchor Browser. Cloudflare's own Browser Rendering signs requests. Cloudflare said enterprise customers would soon be able to act on signed agents as a group.
- Cloudflare's post on Perplexity says "ChatGPT Agent is signing http requests using the newly proposed open standard Web Bot Auth". Separately, the "Forget IPs" post quotes an OpenAI engineer that OpenAI signs all Operator requests with RFC 9421.

Cloudflare sells the bot-management products that consume these signatures; the signed-agent and verified-bot classifications are its own policy.

## Key Parameters

| Item | Value |
|---|---|
| Signing standard | RFC 9421 HTTP Message Signatures |
| Key type (Cloudflare) | Ed25519, public key as JWK, `keyid` = base64url JWK thumbprint |
| `tag` (requests) | `web-bot-auth` |
| `tag` (directory response, Cloudflare docs) | `http-message-signatures-directory` |
| Directory path | `/.well-known/http-message-signatures-directory` |
| Directory media type | `application/http-message-signatures-directory+json` |
| Required request headers | `Signature-Input`, `Signature`, `Signature-Agent` |
| Reference code | `cloudflare/web-bot-auth` repo: browser extension, Rust, Workers, Caddy plugin examples |

### Visa Trusted Agent Protocol (TAP)

Visa announced TAP on 17 October 2025. Its press release says it is "built upon the foundational HTTP Message Signature standard, and aligned with Web Bot Auth", citing Adobe Data Insights (August 2025) for a 4,700% rise in AI-driven visits to US retail sites (Adobe's measure of AI-driven visit share; Visa's framing). Visa's sample repository implements RFC 9421 with Ed25519 and RSA-PSS-SHA256, signature parameters including `created`, `expires`, `nonce`, `keyId` and `tag`, and two tags: `agent-browser-auth` (product details) and `agent-payer-auth` (checkout). TAP carries merchant-facing agent and consumer identifiers in addition to identity; see [[concepts/agentic-commerce-protocols]].

## When To Use

- **You operate an agent or crawler** that sites block by default: sign requests, publish a directory, register with the CDNs you care about.
- **You run a site** and want to allow specific agents past bot rules without maintaining IP lists: verify signatures at the edge, typically through a CDN that already does it.
- **You sell through agents**: TAP-style verification is aimed at merchants distinguishing agent checkout from fraud.
- Cloudflare's readiness scanner says not every site needs to implement it: "If your site just serves content, and doesn't make requests to other sites, you don't need it." That applies to the signing side.

## Risks & Pitfalls

- **Draft status.** The protocol is an Internet-Draft; field names and header forms have already changed (bare string to dictionary `Signature-Agent`).
- **Signature is not authorization.** The draft says an outer signature "does not express authorization, delegation, or consent". Verified identity does not tell you whether the agent may take an action.
- **Replays.** A signature can be replayed within its validity window; Cloudflare recommends short `expires`.
- **Intermediaries and transitive trust.** An operator signing for many end users is trusted per operator, not per user (Cloudflare verified-bots docs).
- **Coverage is partial.** Many agents do not sign. Cloudflare reported that Perplexity's undeclared "stealth" crawler presented a generic Chrome-on-macOS user agent (`Chrome/124.0.0.0`) alongside the declared `Perplexity-User/1.0`; the dispute is covered in [[concepts/user-triggered-fetchers-and-robots]]. Whether specific browser agents (Atlas, Comet, Gemini auto browse) sign requests is not established in the gathered sources.
- **Vendor lock of semantics.** Verified-bot and signed-agent lists are Cloudflare's, with its own criteria.

## Related Concepts

- [[concepts/ai-crawlers-and-user-agents]] — UA and IP verification, the older methods.
- [[concepts/paid-access-x402-and-pay-per-crawl]] — pay per crawl uses signed requests to identify payers.
- [[concepts/agent-discovery-files]] — key directories are one of the `.well-known` artifacts.
- [[entities/cloudflare]]

## Sources

- `raw/web_community-rfc-9421-http-message-signatures-rfc-editor.md` — RFC 9421 date and role.
- `raw/web_community-draft-ietf-webbotauth-httpsig-protocol-01.md` — draft -01 dates, status, authors.
- `raw/github_doc-draft-ietf-webbotauth-httpsig-protocol-md.md` — draft text: `Signature-Agent`, tag, directory, replay.
- `raw/web_community-draft-meunier-web-bot-auth-architecture-05.md` — Informational architecture draft and expiry.
- `raw/web_community-forget-ips-using-cryptography-to-verify-bot-and-agent-traffi.md` — problem statement, OpenAI quote, example headers.
- `raw/web_community-the-age-of-agents-cryptographically-recognizing-agent-traffi.md` — signed agents and first cohort.
- `raw/web_community-web-bot-auth-cloudflare-bot-solutions-docs.md` — setup requirements and header table.
- `raw/web_community-verified-bots-cloudflare-bot-solutions-docs-2.md` — verified-bot criteria and labels.
- `raw/web_community-perplexity-is-using-stealth-undeclared-crawlers-to-evade-web.md` — ChatGPT Agent signing claim; Chrome UA example.
- `raw/github_doc-readme-md-13.md` — Cloudflare reference implementations.
- `raw/web_community-visa-introduces-trusted-agent-protocol-an-ecosystem-led-fram.md` — TAP announcement, Web Bot Auth alignment.
- `raw/github_doc-tap-agent-readme-md.md` — TAP signature algorithms and tags.
- `raw/github_doc-readme-md-14.md` — TAP merchant-facing goals.
- `raw/web_community-introducing-the-agent-readiness-score-check-to-see-if-your-s.md` — "you don't need it" guidance.