# Agentic SEO — full corpus # LLM Wiki An open-source template for building LLM-powered knowledge bases, following [Andrej Karpathy's "LLM Wiki" pattern](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f). You provide raw sources. The LLM reads them, writes structured wiki pages, cross-links everything, and maintains it over time. You never edit the wiki directly — you curate sources and ask questions. ## How It Works The system has three layers: ``` raw/ Sources you collect (articles, transcripts, notes, PDFs) wiki/ LLM-written & maintained pages (summaries, concepts, entities, syntheses) CLAUDE.md Schema that tells the LLM how to structure everything ``` Three operations drive the workflow: | Operation | Trigger | What happens | |-----------|---------|--------------| | **Ingest** | "ingest raw/my-source.txt" | LLM reads the source, creates a summary page, creates/updates concept and entity pages, adds cross-links, updates the index and log | | **Query** | Ask any question | LLM searches the wiki, synthesizes an answer with citations, optionally creates a synthesis page for novel insights | | **Lint** | "lint" or "health check" | LLM audits all pages for orphans, contradictions, missing links, incomplete sections, and low-confidence claims — fixes what it can, reports the rest | ## Quick Start 1. **Clone this repo** ```bash git clone https://github.com/YOUR_USERNAME/llm-wiki.git my-knowledge-base cd my-knowledge-base ``` 2. **Customize CLAUDE.md** for your domain - Update the Purpose section with your topic - Replace the placeholder tagging taxonomy with your own categories - Adjust confidence level descriptions if needed - Everything else (workflows, page formats, linking rules) works as-is 3. **Drop sources into `raw/`** - Text files, transcripts, articles, notes — any plain text - These are immutable once added; the LLM never modifies them 4. **Tell the LLM to ingest** ``` ingest raw/my-first-source.txt ``` The LLM will create summary pages, concept pages, entity pages, cross-links, and update the index. 5. **Ask questions** ``` What are the key differences between X and Y? ``` The LLM answers from the wiki, citing specific pages. 6. **Run health checks** ``` lint ``` The LLM audits the wiki and fixes issues. ## Directory Structure ``` . ├── CLAUDE.md # Schema — the LLM's instructions ├── raw/ # Your source documents (immutable) └── wiki/ ├── index.md # Master catalog of all pages ├── log.md # Append-only activity log ├── dashboard.md # Dataview dashboard (Obsidian) ├── analytics.md # Charts View analytics (Obsidian) ├── flashcards.md # Spaced repetition cards ├── summaries/ # One page per source document ├── concepts/ # Concept and framework pages ├── entities/ # People, tools, organizations, etc. ├── syntheses/ # Cross-cutting analyses and comparisons ├── journal/ # Research/session journal entries │ └── template.md # Journal entry template └── presentations/ # Marp slide decks ``` ## Enhancements This template includes several extras beyond the core wiki pattern: ### Dataview Dashboard (`wiki/dashboard.md`) Live queries that surface low-confidence pages, recent updates, concepts by tag, and pages with the most sources. Requires the [Dataview](https://github.com/blacksmithgu/obsidian-dataview) Obsidian plugin. ### Charts View Analytics (`wiki/analytics.md`) Visual analytics with pie charts, bar charts, and word clouds. Requires the [Charts View](https://github.com/caronchen/obsidian-chartsview-plugin) Obsidian plugin. ### Mermaid Diagrams Use Mermaid code blocks in any wiki page to create flowcharts, sequence diagrams, or concept maps. Native support in Obsidian and GitHub. ### Marp Slides (`wiki/presentations/`) Create slide decks from markdown using [Marp](https://marp.app/). Drop presentation files in this directory. ### Research Journal (`wiki/journal/`) Track your research sessions, experiments, or applied work with the included template. The LLM can reference journal entries when answering queries. ### Spaced Repetition (`wiki/flashcards.md`) Flashcards in the format used by the [Spaced Repetition](https://github.com/st3v3nmw/obsidian-spaced-repetition) Obsidian plugin. Ask the LLM to generate flashcards from any wiki page. ### MCP Server This repo works with Claude Code's MCP server capabilities. Point an MCP-compatible client at this repo and the LLM can read/write the wiki programmatically. ## Customizing for Your Domain The schema in `CLAUDE.md` is domain-agnostic. To adapt it: 1. **Purpose** — Describe your knowledge domain in one paragraph 2. **Tagging taxonomy** — Replace placeholder categories with your own (e.g., for a cooking KB: `cuisine`, `technique`, `ingredient`, `equipment`) 3. **Confidence levels** — Adjust the descriptions to match your domain's evidence standards 4. **Entity types** — Update the entity page description to match what entities mean in your domain (people, tools, companies, etc.) 5. **Journal template** — Customize `wiki/journal/template.md` for your workflow Everything else — page format, linking conventions, workflows, rules — is universal and works across domains. ## Example Domains This template works for any knowledge-intensive topic: - **Research notes** — papers, experiments, methodologies - **Book analysis** — themes, characters, author techniques - **Competitive analysis** — companies, products, market trends - **Course notes** — lectures, readings, key concepts - **Personal development** — frameworks, habits, book summaries - **Technical documentation** — APIs, architectures, design patterns - **Hobby deep-dives** — any subject you want to master ## License MIT --- title: "Agentic SEO KB — Master Index" type: index updated: 2026-11-04 --- # Agentic SEO KB — Master Index **Domain:** Agentic SEO as three problems: (A) getting cited or recommended in AI answers (Google AI Overviews and AI Mode, ChatGPT search, Perplexity, Copilot, Claude); (B) being legible and cheap for agents that fetch pages (crawler user-agents, robots.txt, Content Signals, llms.txt, markdown negotiation, JavaScript rendering, structured data); (C) being operable by agents that act for users (browser agents, WebMCP, discovery files, Web Bot Auth, x402 and pay per crawl, agentic-commerce protocols). **Scope:** Official vendor docs, specs, RFCs and IETF drafts, protocol repositories, independent and vendor-run studies, and trade coverage, all fetched 2026-11-04. Not covered: the MCP protocol itself (see the MCP wiki), Shopify developer APIs, paid search ads, and classic SEO except where official AI-search guidance says it applies. **Pages:** 37 (5 syntheses · 19 concepts · 9 entities · 4 summaries) **Sources fetched:** 2026-11-04 (416 raw files from 88 hosts and repositories; see [[summaries/source-map]]) Start at [[syntheses/overview]] if you are new. Most numbers in this field come from vendors that sell SEO, visibility or bot-control products; pages name who measured each one. ## Syntheses - [[syntheses/overview]] — What agentic SEO covers, who the actors are, what is established vs contested vs emerging as of 2026-11, and where to start for each problem. - [[syntheses/agent-readiness-checklist]] — Prioritized to-do list across citation, legibility and operability, with the evidence strength and cost of each item and notes by site type. - [[syntheses/myths-vs-evidence]] — Common claims (llms.txt gets you cited, schema boosts citations, AI crawlers render JS, robots.txt blocks agents…) against what was measured, by whom, with a verdict. - [[syntheses/serving-agents-different-content]] — Markdown negotiation, .md mirrors, bot-only pages, CDN-converted markdown and llms.txt, measured against Google's and Bing's statements on cloaking. - [[syntheses/crawler-access-policy]] — Which bots to allow or block for each goal (be cited, avoid training, get paid, protect checkout), with verbatim robots.txt and Content-Signal examples and their limits. ## Concepts ### A. Cited in AI answers - [[concepts/ai-search-retrieval-and-fan-out]] — How answer engines retrieve and pick citations: RAG over a search index, query fan-out, and how Google, ChatGPT, Perplexity and Copilot differ. - [[concepts/generative-engine-optimization]] — GEO/AEO defined; what evidence says drives citation, what Google says to ignore, and where the vendor studies disagree. - [[concepts/ai-search-controls-and-reporting]] — nosnippet, data-nosnippet, max-snippet, Google's Search generative AI control, Google-Extended, and Google's and Bing's AI reporting. - [[concepts/ai-visibility-measurement]] — How prompt tracking works (UI vs API, sampling, variance), first-party data, and how to read share-of-voice claims. - [[concepts/content-manipulation-and-prompt-injection]] — Adversarial GEO, AI recommendation poisoning, hidden prompts against browsing agents, spam policies, and the evidence on what works. ### B. Legible to fetching agents - [[concepts/ai-crawlers-and-user-agents]] — Every vendor's user-agent tokens by purpose (training, search index, user-triggered, agent) and how to verify them. - [[concepts/user-triggered-fetchers-and-robots]] — Why user-triggered fetchers treat robots.txt differently, each vendor's stated behaviour, measured non-compliance, and the Cloudflare–Perplexity dispute. - [[concepts/robots-txt-for-ai]] — RFC 9309 mechanics, per-purpose recipes, Google-Extended and Applebot-Extended semantics, CDN-managed robots.txt and pitfalls. - [[concepts/content-signals-and-aipref]] — Cloudflare Content Signals (search, ai-input, ai-train), the IETF AI-preferences drafts and their status, and who honours them. - [[concepts/llms-txt]] — The llms.txt proposal, adoption, evidence of use, and Google Search's stance vs Chrome Lighthouse's audit. - [[concepts/markdown-content-negotiation]] — Serving markdown on `Accept: text/markdown` or `.md` URLs, measured token savings, which agents ask, and consistency concerns. - [[concepts/js-rendering-by-ai-crawlers]] — Which crawlers and fetchers execute JavaScript, per study and date, and what that means for client-rendered sites. - [[concepts/structured-data-and-ai]] — JSON-LD and schema.org in AI search: official statements, measured effects, and where markup still matters. ### C. Operable by acting agents - [[concepts/agent-perception-and-site-ux]] — How browser agents perceive pages (screenshots, DOM, accessibility tree) and the UX guidance that follows. - [[concepts/webmcp]] — The proposed browser API for pages to register agent tools: imperative vs declarative APIs, origin-trial status, naming, security. - [[concepts/agent-discovery-files]] — MCP server cards and ai-catalog, RFC 9727 api-catalog, RFC 8288 Link headers: status, conflicting paths and adoption. - [[concepts/agent-identity-web-bot-auth]] — Signed agent requests: HTTP Message Signatures (RFC 9421), the Web Bot Auth drafts, key directories and Cloudflare signed agents. - [[concepts/paid-access-x402-and-pay-per-crawl]] — HTTP 402 payment flows: the x402 protocol and Cloudflare pay per crawl, and how they relate. - [[concepts/agentic-commerce-protocols]] — ACP, UCP, AP2 and Visa TAP compared: what each covers, status, and what a merchant actually implements. ## Entities - [[entities/google-search]] — AI Overviews, AI Mode, Google-Extended, Google-Agent and Gemini in Chrome, with pointers to Google's official guidance. - [[entities/openai]] — GPTBot, OAI-SearchBot and ChatGPT-User; ChatGPT search; ChatGPT agent and Atlas; commerce and Instant Checkout status. - [[entities/anthropic]] — ClaudeBot, Claude-SearchBot and Claude-User, the API web search tool, and Claude in Chrome. - [[entities/perplexity]] — PerplexityBot, Perplexity-User, Comet, and both sides of the stealth-crawling dispute. - [[entities/microsoft-bing-copilot]] — Bing's grounding index, publisher guidance for Copilot, the AI Performance report, and Microsoft's poisoning research. - [[entities/cloudflare]] — AI Crawl Control, managed robots.txt, Content Signals, Markdown for Agents, pay per crawl, signed agents, isitagentready and Radar. - [[entities/ai-visibility-tools]] — Profound, Peec AI, Otterly, Ahrefs Brand Radar, Semrush and Scrunch: what each measures and how much they disclose. - [[entities/agentic-commerce-protocol-acp]] — OpenAI and Stripe's ACP: dated spec versions, checkout and delegated-payment flows, feeds, RFCs and unreleased changes. - [[entities/universal-commerce-protocol-ucp]] — Google's UCP and AP2: capabilities, mandates, transports, versioning, and how the two fit together. ## Summaries - [[summaries/geo-research-papers]] — The GEO research literature: the Princeton GEO paper in detail, GEO-Bench, attack papers and the critique of their evaluations. - [[summaries/ai-citation-studies]] — One row per industry citation study: publisher, date, sample, finding and vendor interest. Numbers are never merged across rows. - [[summaries/bot-and-agent-traffic-reports]] — Cloudflare, TollBit, Fastly and others: the population each report counts and its headline numbers with dates. - [[summaries/source-map]] — Every source host and repository by class (official, study, coverage, spec repo), how many files pages cite, and what was not gathered. --- title: "Agent Discovery Files" type: concept tags: [well-known, ai-catalog, mcp-server-card, rfc-9727, rfc-8288, link-headers, discovery] created: 2026-11-04 updated: 2026-11-04 confidence: medium sources: ["raw/github_doc-readme-md-10.md", "raw/github_doc-docs-discovery-md.md", "raw/github_doc-docs-core-maintainer-roadmap-md.md", "raw/web_community-sep-2127-mcp-server-cards-http-server-discovery-model-contex.md", "raw/web_community-rfc-9727-api-catalog-a-well-known-uri-and-link-relation-to-h.md", "raw/web_community-rfc-8288-web-linking-rfc-editor.md", "raw/web_community-introducing-the-agent-readiness-score-check-to-see-if-your-s.md", "raw/web_community-is-your-site-agent-ready.md", "raw/web_community-draft-serra-mcp-discovery-uri-04.md", "raw/github_doc-docs-documentation-core-concepts-md.md"] --- # Agent Discovery Files ## Definition Discovery files are small machine-readable documents or HTTP headers a site publishes at predictable locations so an agent can find out what the site offers (APIs, MCP servers, skills, commerce endpoints) before it interacts. They answer "where do I connect?" the way `robots.txt` answers "what may I fetch?". This page covers the agent-capability layer; crawl rules and AI-usage preferences are in [[concepts/robots-txt-for-ai]] and [[concepts/content-signals-and-aipref]], and site summaries for LLMs in [[concepts/llms-txt]]. Most items below are drafts or experiments. Status is given per item; do not read presence in a scanner as evidence of a standard. ## How It Works Agents try fixed paths (usually under `/.well-known/`, RFC 8615) or read an HTTP `Link` header, then follow URLs in what they find. ### RFC 9727: api-catalog (Internet Standards Track) - Well-known URI: `/.well-known/api-catalog`. A publisher must resolve an HTTPS `GET` to it with an API catalog document, and an HTTPS `HEAD` with a `Link` header carrying the relations the RFC defines. - The document must be in Linkset format, `application/linkset+json` (RFC 9264), and should carry a `profile` parameter with the value `https://www.rfc-editor.org/info/rfc9727`. - It also defines the `api-catalog` link relation, which a page or response can use to point at the catalog wherever it lives. - It does not mandate where API endpoints live or what their documentation looks like; the catalog lists them and links to specs, docs and status endpoints. ### RFC 8288: Web Linking (Internet Standards Track, October 2017) The `Link` response header carries typed links in the HTTP response itself, so an agent can find resources without parsing HTML. Cloudflare's readiness post gives this example: ```http HTTP/1.1 200 OK Link: ; rel="api-catalog" ``` ### AI Catalog and MCP Server Cards (draft, with a status conflict) - The MCP experimental extension repo defines an **AI Catalog** served at `/.well-known/ai-catalog.json`, ideally with media type `application/ai-catalog+json`. Clients doing domain-level discovery "SHOULD attempt to retrieve" it. The catalog format itself is maintained in a separate Agent-Card/ai-catalog repository referenced by the MCP docs. - A catalog entry for an MCP server has `identifier`, `type` (must be `application/mcp-server-card+json`), and one of `url` or `data`. Identifiers for open systems use `urn:air:{publisher}:{namespace}:{name}`. - A **Server Card** describes a remote MCP server: `name` (reverse-DNS, e.g. `com.example/weather`), version, description, transport endpoints. It intentionally omits tools, resources and prompts, which remain subject to runtime listing. - Reserved default card location: `MAY` host at `GET /server-card`; any unreserved URI is valid, and a catalog entry's `url` gives the exact location. The repo's "alternatives considered" section explicitly does not recommend a `.well-known` path for individual cards. - Clients fetching a card SHOULD send `Accept: application/mcp-server-card+json`. Hosted cards MUST be served over HTTPS (TLS 1.2 or later) in production; HTTP MAY be used for local development only. Hosted card endpoints MUST include CORS headers (`Access-Control-Allow-Origin: *`, `Access-Control-Allow-Methods: GET`, `Access-Control-Allow-Headers: Content-Type, If-None-Match`, `Access-Control-Expose-Headers: ETag`). Caching headers are SHOULD: `Cache-Control: public, max-age=3600` plus an `ETag`, with `304 Not Modified` on a matching `If-None-Match`. - A card is advisory: clients "MUST NOT treat Server Card contents as authoritative for security or access-control decisions" and should prefer live `server/discover` values when they disagree. ## Key Parameters | Artifact | Path / location | Status as the source states it | |---|---|---| | API catalog | `/.well-known/api-catalog` | RFC 9727, Internet Standards Track | | Link header | `Link: <...>; rel="api-catalog"` | RFC 8288, Standards Track | | AI Catalog | `/.well-known/ai-catalog.json` | Draft ("Protocol Revision: draft" in MCP docs) | | MCP Server Card | `/server-card` | See conflict below | | Cloudflare's example card | `/.well-known/mcp/server-card.json` | Cloudflare calls it "a proposal currently in draft" | | MCP discovery URI draft | `/.well-known/mcp-server`, `mcp://` scheme | draft-serra-mcp-discovery-uri-04, Intended status: Informational, dated 25 March 2026, expires 25 September 2026 | | Agent Skills index | `.well-known/agent-skills/index.json` | Cloudflare: "We've proposed" | | HTTP signature keys | `/.well-known/http-message-signatures-directory` | See [[concepts/agent-identity-web-bot-auth]] | | UCP profile | `/.well-known/ucp` | Part of Google's UCP; see [[concepts/agentic-commerce-protocols]] | ### Conflict: are Server Cards accepted? - The MCP site's SEP-2127 page shows status **Final**, type Extensions Track, created 2026-01-21, with a note that it is a historical record and the detailed wire format is maintained in the experimental repo. - The experimental repo's README says "Status: Experimental... not an accepted or official MCP extension" and that the SEP "is still under review". Its roadmap file says SEP-2127 "is in review" and that the Extensions Track also needs a merged SDK reference implementation. Both were fetched on the same day; the pages may have been updated at different times. The SEP page itself says Final status is "preserved as a historical record of the design as accepted" and defers to the current specification; no gathered source reconciles this with the repo's Experimental label. ### Conflict: where is the card? Cloudflare's Agent Readiness post (and isitagentready.com's checker) uses `/.well-known/mcp/server-card.json` with a `$schema` of `https://static.modelcontextprotocol.io/schemas/mcp-server-card/v1.json`, and a card shape that includes `serverInfo`, `transport` and a `tools` array. The MCP repo's discovery doc uses the AI Catalog plus `/server-card`, and the card schema omits tool listings. These are different shapes of the same idea; the MCP-hosted documents are the more authoritative source on the MCP side. ### What scanners check (isitagentready.com) The scanner lists: robots.txt, Sitemap, Link headers, DNS for AI Discovery (DNS-AID); Markdown negotiation; AI bot rules, Content Signals, Web Bot Auth; API Catalog, OAuth discovery, OAuth Protected Resource, Auth.md, MCP Server Card, A2A Agent Card, Agent Skills, WebMCP, ARD manifest, Personal Agent Protocol (Poppy); and commerce (x402, MPP, UCP, ACP). Cloudflare's launch post scores four dimensions and says commerce checks "do not currently count towards the score". Cloudflare sells bot-control products and runs the scanner; the checklist is its view of the ecosystem. ## When To Use - You operate a public API: publish the RFC 9727 catalog, the only dedicated catalog format here that is a finished RFC (RFC 8288 Link headers are also Standards Track). - You run a remote MCP server and want clients and registries to find it without manual configuration: an AI Catalog entry pointing at a Server Card, accepting that the format may change. - You already emit `Link` headers: adding the `api-catalog` relation costs little. ## Risks & Pitfalls - **Adoption is near zero.** Cloudflare Radar scanned the 200,000 most visited domains (after filtering categories such as redirects and ad servers) and reported, as of its launch post: `robots.txt` on 78% of sites, Content Signals on 4%, `Accept: text/markdown` handling on 3.9%, and MCP Server Cards plus RFC 9727 API Catalogs "together on fewer than 15 sites". This is a vendor-run scan of one population, not agent behaviour; it says nothing about whether any agent reads these files. - **No evidence of agent use in the gathered sources.** Wiki analysis: the sources document publication and scanner support, not fetch rates by agents. - **Churn.** The card path, schema URL and shape differ across sources; the AI Catalog and Server Card specs are drafts; the discovery URI draft expires 25 September 2026 (before 2026-11-04; no later revision is in the gathered sources). - **Information disclosure.** Public catalogs must not contain credentials, internal topology or proprietary logic (MCP discovery doc). A card can also mislead a client; the MCP docs describe stale or crafted cards as "a mild confusion or downgrade vector". - **Scanner scores are not standards compliance.** The checker mixes RFCs, drafts and one vendor's own proposals. ## Related Concepts - [[concepts/webmcp]] — in-page tools, which agents can only find by visiting the page. - [[concepts/agent-identity-web-bot-auth]] - [[concepts/llms-txt]] — human/LLM-readable site summary, a different kind of discovery. - [[entities/cloudflare]] ## Sources - `raw/github_doc-readme-md-10.md` — MCP Server Card extension: experimental status, scope, `$schema` convention. - `raw/github_doc-docs-discovery-md.md` — AI Catalog path and entry fields, card location, headers, security. - `raw/github_doc-docs-core-maintainer-roadmap-md.md` — SEP-2127 "in review" and the roadmap themes. - `raw/web_community-sep-2127-mcp-server-cards-http-server-discovery-model-contex.md` — SEP page showing Final status. - `raw/web_community-rfc-9727-api-catalog-a-well-known-uri-and-link-relation-to-h.md` — RFC 9727 requirements. - `raw/web_community-rfc-8288-web-linking-rfc-editor.md` — RFC 8288 status and Link header examples. - `raw/web_community-introducing-the-agent-readiness-score-check-to-see-if-your-s.md` — Cloudflare scanner, Radar adoption numbers, example paths. - `raw/web_community-is-your-site-agent-ready.md` — the scanner's current check list. - `raw/web_community-draft-serra-mcp-discovery-uri-04.md` — `/.well-known/mcp-server` and `mcp://` draft. - `raw/github_doc-docs-documentation-core-concepts-md.md` — UCP profile at `/.well-known/ucp`. --- title: "Agent Identity and Web Bot Auth" type: concept tags: [web-bot-auth, http-message-signatures, rfc-9421, signed-agents, verified-bots, visa-tap, ietf] created: 2026-11-04 updated: 2026-11-04 confidence: medium sources: ["raw/web_community-rfc-9421-http-message-signatures-rfc-editor.md", "raw/web_community-draft-ietf-webbotauth-httpsig-protocol-01.md", "raw/github_doc-draft-ietf-webbotauth-httpsig-protocol-md.md", "raw/web_community-draft-meunier-web-bot-auth-architecture-05.md", "raw/web_community-forget-ips-using-cryptography-to-verify-bot-and-agent-traffi.md", "raw/web_community-the-age-of-agents-cryptographically-recognizing-agent-traffi.md", "raw/web_community-web-bot-auth-cloudflare-bot-solutions-docs.md", "raw/web_community-verified-bots-cloudflare-bot-solutions-docs-2.md", "raw/web_community-perplexity-is-using-stealth-undeclared-crawlers-to-evade-web.md", "raw/github_doc-readme-md-13.md", "raw/web_community-visa-introduces-trusted-agent-protocol-an-ecosystem-led-fram.md", "raw/github_doc-tap-agent-readme-md.md", "raw/github_doc-readme-md-14.md", "raw/web_community-introducing-the-agent-readiness-score-check-to-see-if-your-s.md"] --- # Agent Identity and Web Bot Auth ## Definition Web Bot Auth is a way for automated clients to prove who they are by signing their HTTP requests, so a site can verify the sender without trusting a `User-Agent` string or an IP list. It builds on HTTP Message Signatures (RFC 9421, February 2024) and adds three things: a `Signature-Agent` header that says where the sender's public keys live, a key-directory format based on JSON Web Key Sets, and a well-known URI for that directory. It is an IETF work item, not a finished standard. It answers "who is this agent?", not "may it do this?"; permission lives in [[concepts/robots-txt-for-ai]] and [[concepts/content-signals-and-aipref]]. ## How It Works Cloudflare's "Forget IPs" post states the problem with the older mechanisms: `User-Agent` is easily spoofed and agents often send the Chrome user agent; IP ranges are shared and change; per-site bearer tokens do not scale. See [[concepts/ai-crawlers-and-user-agents]] for those methods. The flow: 1. The agent holds a key pair and publishes the public key (as a JWK) in a key directory. Cloudflare's setup guide uses Ed25519; the draft's `keyid` thumbprint rules also cover RSA and EC keys. 2. Per request, it sends `Signature-Input`, `Signature` and `Signature-Agent`. The signature must cover `@authority` (target host) or `@target-uri`, plus the request's `Signature-Agent` member. 3. The verifier resolves the `Signature-Agent` URL, fetches keys, selects the one matching `keyid`, and checks the signature, `created` and `expires`. Current IETF working-group draft (`draft-ietf-webbotauth-httpsig-protocol-01`, published 29 October 2026, Intended Status Standards Track, expires 2 May 2027; authors from Cloudflare and Google) requires: - `tag` must be `web-bot-auth`; `expires` recommended no more than 24 hours after `created`. - `Signature-Agent` is a Dictionary Structured Header whose members are `https` URIs; covered component form `("@authority" "signature-agent";key="sig1")`. Earlier drafts defined it as a bare string and "deployments still send it"; signers must send the dictionary form, verifiers may accept the legacy form. - A signed request must carry `Signature-Agent`. Replay protection (including `nonce`) is left to RFC 9421; the draft adds no nonce requirement. - The identifier of an agent is the URL, not the key, so keys can rotate. - Directories are JWKS served over HTTPS; at the well-known URI the media type is `application/http-message-signatures-directory+json`. ```http GET /path/to/resource HTTP/1.1 Host: origin.example.com Signature: sig=abc== Signature-Input: sig=("@authority" "signature-agent";key="sig");created=1700000000;expires=1700011111;keyid="ba3e64==";tag="web-bot-auth" Signature-Agent: sig="https://signer.example.com" ``` (Example from the draft, line-wrapping removed; values are illustrative.) An older companion, `draft-meunier-web-bot-auth-architecture-05`, is Intended Status Informational and states it "will expire on 3 September 2026". Cloudflare's docs still reference `draft-meunier-http-message-signatures-directory-03` and the architecture draft revision 02, so vendor docs lag the working-group text. ### Cloudflare's implementation - Docs (updated 2026-10-08) require hosting the key directory at `/.well-known/http-message-signatures-directory`, signing the directory response with `tag` set to `http-message-signatures-directory`, and registering the bot via the dashboard Bot Submission Form (Verification Method: Request Signature). Cloudflare accepts Ed25519 keys. - **Verified bots** (docs updated 2026-11-02): must self-identify deterministically (Web Bot Auth signature, published IP list with stable user-agent, or reverse DNS) and must obey `robots.txt` and crawl directives. Behaviours include Search, Agent, Training and Transact; bots are labelled Direct or Intermediary. Cloudflare says it is experimenting with forwarding end-user information via the `Forwarded` header (RFC 7239). - **Signed agents** (post undated in the page; its Radar caption covers 21-27 August 2025, so late August 2025 is inferred): an extension of verified bots for agents directed by end users via remote browsing infrastructure. First cohort named: ChatGPT agent, Goose (Block), Browserbase, Anchor Browser. Cloudflare's own Browser Rendering signs requests. Cloudflare said enterprise customers would soon be able to act on signed agents as a group. - Cloudflare's post on Perplexity says "ChatGPT Agent is signing http requests using the newly proposed open standard Web Bot Auth". Separately, the "Forget IPs" post quotes an OpenAI engineer that OpenAI signs all Operator requests with RFC 9421. Cloudflare sells the bot-management products that consume these signatures; the signed-agent and verified-bot classifications are its own policy. ## Key Parameters | Item | Value | |---|---| | Signing standard | RFC 9421 HTTP Message Signatures | | Key type (Cloudflare) | Ed25519, public key as JWK, `keyid` = base64url JWK thumbprint | | `tag` (requests) | `web-bot-auth` | | `tag` (directory response, Cloudflare docs) | `http-message-signatures-directory` | | Directory path | `/.well-known/http-message-signatures-directory` | | Directory media type | `application/http-message-signatures-directory+json` | | Required request headers | `Signature-Input`, `Signature`, `Signature-Agent` | | Reference code | `cloudflare/web-bot-auth` repo: browser extension, Rust, Workers, Caddy plugin examples | ### Visa Trusted Agent Protocol (TAP) Visa announced TAP on 17 October 2025. Its press release says it is "built upon the foundational HTTP Message Signature standard, and aligned with Web Bot Auth", citing Adobe Data Insights (August 2025) for a 4,700% rise in AI-driven visits to US retail sites (Adobe's measure of AI-driven visit share; Visa's framing). Visa's sample repository implements RFC 9421 with Ed25519 and RSA-PSS-SHA256, signature parameters including `created`, `expires`, `nonce`, `keyId` and `tag`, and two tags: `agent-browser-auth` (product details) and `agent-payer-auth` (checkout). TAP carries merchant-facing agent and consumer identifiers in addition to identity; see [[concepts/agentic-commerce-protocols]]. ## When To Use - **You operate an agent or crawler** that sites block by default: sign requests, publish a directory, register with the CDNs you care about. - **You run a site** and want to allow specific agents past bot rules without maintaining IP lists: verify signatures at the edge, typically through a CDN that already does it. - **You sell through agents**: TAP-style verification is aimed at merchants distinguishing agent checkout from fraud. - Cloudflare's readiness scanner says not every site needs to implement it: "If your site just serves content, and doesn't make requests to other sites, you don't need it." That applies to the signing side. ## Risks & Pitfalls - **Draft status.** The protocol is an Internet-Draft; field names and header forms have already changed (bare string to dictionary `Signature-Agent`). - **Signature is not authorization.** The draft says an outer signature "does not express authorization, delegation, or consent". Verified identity does not tell you whether the agent may take an action. - **Replays.** A signature can be replayed within its validity window; Cloudflare recommends short `expires`. - **Intermediaries and transitive trust.** An operator signing for many end users is trusted per operator, not per user (Cloudflare verified-bots docs). - **Coverage is partial.** Many agents do not sign. Cloudflare reported that Perplexity's undeclared "stealth" crawler presented a generic Chrome-on-macOS user agent (`Chrome/124.0.0.0`) alongside the declared `Perplexity-User/1.0`; the dispute is covered in [[concepts/user-triggered-fetchers-and-robots]]. Whether specific browser agents (Atlas, Comet, Gemini auto browse) sign requests is not established in the gathered sources. - **Vendor lock of semantics.** Verified-bot and signed-agent lists are Cloudflare's, with its own criteria. ## Related Concepts - [[concepts/ai-crawlers-and-user-agents]] — UA and IP verification, the older methods. - [[concepts/paid-access-x402-and-pay-per-crawl]] — pay per crawl uses signed requests to identify payers. - [[concepts/agent-discovery-files]] — key directories are one of the `.well-known` artifacts. - [[entities/cloudflare]] ## Sources - `raw/web_community-rfc-9421-http-message-signatures-rfc-editor.md` — RFC 9421 date and role. - `raw/web_community-draft-ietf-webbotauth-httpsig-protocol-01.md` — draft -01 dates, status, authors. - `raw/github_doc-draft-ietf-webbotauth-httpsig-protocol-md.md` — draft text: `Signature-Agent`, tag, directory, replay. - `raw/web_community-draft-meunier-web-bot-auth-architecture-05.md` — Informational architecture draft and expiry. - `raw/web_community-forget-ips-using-cryptography-to-verify-bot-and-agent-traffi.md` — problem statement, OpenAI quote, example headers. - `raw/web_community-the-age-of-agents-cryptographically-recognizing-agent-traffi.md` — signed agents and first cohort. - `raw/web_community-web-bot-auth-cloudflare-bot-solutions-docs.md` — setup requirements and header table. - `raw/web_community-verified-bots-cloudflare-bot-solutions-docs-2.md` — verified-bot criteria and labels. - `raw/web_community-perplexity-is-using-stealth-undeclared-crawlers-to-evade-web.md` — ChatGPT Agent signing claim; Chrome UA example. - `raw/github_doc-readme-md-13.md` — Cloudflare reference implementations. - `raw/web_community-visa-introduces-trusted-agent-protocol-an-ecosystem-led-fram.md` — TAP announcement, Web Bot Auth alignment. - `raw/github_doc-tap-agent-readme-md.md` — TAP signature algorithms and tags. - `raw/github_doc-readme-md-14.md` — TAP merchant-facing goals. - `raw/web_community-introducing-the-agent-readiness-score-check-to-see-if-your-s.md` — "you don't need it" guidance. --- title: "Agent Perception and Site UX" type: concept tags: [browser-agents, accessibility-tree, semantic-html, ux, lighthouse] created: 2026-11-04 updated: 2026-11-04 confidence: medium sources: ["raw/web_community-build-agent-friendly-websites-web-dev.md", "raw/web_community-how-ai-agents-see-your-website-and-how-to-build-for-them.md", "raw/web_community-building-browser-agents-architecture-security-and-practical-.md", "raw/web_community-google-s-guide-to-optimizing-for-generative-ai-features-on-g.md", "raw/web_community-google-checks-websites-for-llms-txt-in-new-agentic-browsing-.md", "raw/web_community-llms-txt-lighthouse-chrome-for-developers.md", "raw/web_community-ask-gemini-in-chrome-to-complete-tasks-for-you-with-auto-bro.md", "raw/web_community-use-claude-in-chrome-safely-claude-help-center.md", "raw/web_community-claude-in-chrome-permissions-guide-claude-help-center.md"] --- # Agent Perception and Site UX ## Definition Browser agents are AI systems that operate a website on a user's behalf: they plan, click, type and submit. They do not look at a monitor. They act on a machine-readable representation of the page, and the quality of that representation determines how well they perform. This page covers how that representation is built and which site-design choices help or hurt. It sits in the "operable by agents" branch of agentic SEO; for the explicit-tool alternative see [[concepts/webmcp]]. ## How It Works web.dev ("Build agent-friendly websites", last updated 2026-04-01) says agents view a site in three primary ways, plus an explicit fourth channel: | Channel | What the agent gets | Strength / weakness (per web.dev) | |---|---|---| | Screenshot | Rendered page analysed by a vision model | Color, size and proximity convey importance; slow and token-expensive, "better as a backup when the structure is confusing" | | Raw HTML / DOM | Nesting, IDs, classes, data strings | Shows relationships (a Buy Now button inside a product container belongs to that product) | | Accessibility tree | Roles, names and states of interactive elements | A "high-fidelity map" that ignores CSS noise | | WebMCP tools | Explicit JS or HTML-annotated tasks | Interpreted "faster and more accurately" than standard actuation; a proposed standard | Single channels leave a "semantic gap": a `
` in the DOM may be a CSS/JS button the agent cannot tell is one, and a screenshot shows where a button sits but not what it triggers. web.dev states that modern agents combine modalities: DOM and accessibility tree for a structured element list, cross-referenced with a visual rendering for layout and grouping. Google's own generative-AI guide (Search Central) describes browser agents the same way: they may analyse screenshots, inspect DOM structure and interpret the accessibility tree. ### Per-product approaches (coverage, not vendor docs) Search Engine Journal (Part 4 of a five-part series) maps platforms to approaches. Treat it as secondary coverage; it quotes OpenAI's Publishers and Developers FAQ as saying ChatGPT Atlas "uses ARIA tags" to interpret page structure. | Product | Reported approach | |---|---| | Anthropic Computer Use | Vision: screenshot, reason, act loop | | Google Project Mariner | Visual plus code structure; reported 83.5% on WebVoyager | | OpenAI Atlas | Accessibility tree (ARIA roles and names) | | OpenAI CUA (Operator, Atlas) | Hybrid: screenshots, DOM, accessibility tree | | Microsoft Playwright MCP | Accessibility snapshots, no screenshots | | Perplexity Comet | Accessibility tree snapshots plus selective vision | A separate arXiv paper (2511.19477, a production browser-agent write-up) reports hybrid accessibility-tree plus selective-vision context and about 85% on WebGames across 53 challenges against about 50% for the best agent the WebGames authors tested (Gemini 2.5 Pro with Browser Use; 95.7% human baseline). It also reports that its October 2025 testing of Atlas behaved like a screenshot-driven agent that struggled with 24px date-picker cells, and that canvas apps (Google Sheets, Figma, Canva) expose no per-element DOM or accessibility nodes. That conflicts with the SEJ table above; Atlas does not publish its internals, and the SEJ article itself describes Atlas both as accessibility-tree-driven and as part of OpenAI's hybrid CUA, so treat these rows as observations, not facts. ## Key Parameters web.dev's recommendations, character-for-character where they name an element: - **Semantic HTML**: use `