--- title: "Agent Discovery Files" type: concept tags: [well-known, ai-catalog, mcp-server-card, rfc-9727, rfc-8288, link-headers, discovery] created: 2026-11-04 updated: 2026-11-04 confidence: medium sources: ["raw/github_doc-readme-md-10.md", "raw/github_doc-docs-discovery-md.md", "raw/github_doc-docs-core-maintainer-roadmap-md.md", "raw/web_community-sep-2127-mcp-server-cards-http-server-discovery-model-contex.md", "raw/web_community-rfc-9727-api-catalog-a-well-known-uri-and-link-relation-to-h.md", "raw/web_community-rfc-8288-web-linking-rfc-editor.md", "raw/web_community-introducing-the-agent-readiness-score-check-to-see-if-your-s.md", "raw/web_community-is-your-site-agent-ready.md", "raw/web_community-draft-serra-mcp-discovery-uri-04.md", "raw/github_doc-docs-documentation-core-concepts-md.md"] --- # Agent Discovery Files ## Definition Discovery files are small machine-readable documents or HTTP headers a site publishes at predictable locations so an agent can find out what the site offers (APIs, MCP servers, skills, commerce endpoints) before it interacts. They answer "where do I connect?" the way `robots.txt` answers "what may I fetch?". This page covers the agent-capability layer; crawl rules and AI-usage preferences are in [[concepts/robots-txt-for-ai]] and [[concepts/content-signals-and-aipref]], and site summaries for LLMs in [[concepts/llms-txt]]. Most items below are drafts or experiments. Status is given per item; do not read presence in a scanner as evidence of a standard. ## How It Works Agents try fixed paths (usually under `/.well-known/`, RFC 8615) or read an HTTP `Link` header, then follow URLs in what they find. ### RFC 9727: api-catalog (Internet Standards Track) - Well-known URI: `/.well-known/api-catalog`. A publisher must resolve an HTTPS `GET` to it with an API catalog document, and an HTTPS `HEAD` with a `Link` header carrying the relations the RFC defines. - The document must be in Linkset format, `application/linkset+json` (RFC 9264), and should carry a `profile` parameter with the value `https://www.rfc-editor.org/info/rfc9727`. - It also defines the `api-catalog` link relation, which a page or response can use to point at the catalog wherever it lives. - It does not mandate where API endpoints live or what their documentation looks like; the catalog lists them and links to specs, docs and status endpoints. ### RFC 8288: Web Linking (Internet Standards Track, October 2017) The `Link` response header carries typed links in the HTTP response itself, so an agent can find resources without parsing HTML. Cloudflare's readiness post gives this example: ```http HTTP/1.1 200 OK Link: ; rel="api-catalog" ``` ### AI Catalog and MCP Server Cards (draft, with a status conflict) - The MCP experimental extension repo defines an **AI Catalog** served at `/.well-known/ai-catalog.json`, ideally with media type `application/ai-catalog+json`. Clients doing domain-level discovery "SHOULD attempt to retrieve" it. The catalog format itself is maintained in a separate Agent-Card/ai-catalog repository referenced by the MCP docs. - A catalog entry for an MCP server has `identifier`, `type` (must be `application/mcp-server-card+json`), and one of `url` or `data`. Identifiers for open systems use `urn:air:{publisher}:{namespace}:{name}`. - A **Server Card** describes a remote MCP server: `name` (reverse-DNS, e.g. `com.example/weather`), version, description, transport endpoints. It intentionally omits tools, resources and prompts, which remain subject to runtime listing. - Reserved default card location: `MAY` host at `GET /server-card`; any unreserved URI is valid, and a catalog entry's `url` gives the exact location. The repo's "alternatives considered" section explicitly does not recommend a `.well-known` path for individual cards. - Clients fetching a card SHOULD send `Accept: application/mcp-server-card+json`. Hosted cards MUST be served over HTTPS (TLS 1.2 or later) in production; HTTP MAY be used for local development only. Hosted card endpoints MUST include CORS headers (`Access-Control-Allow-Origin: *`, `Access-Control-Allow-Methods: GET`, `Access-Control-Allow-Headers: Content-Type, If-None-Match`, `Access-Control-Expose-Headers: ETag`). Caching headers are SHOULD: `Cache-Control: public, max-age=3600` plus an `ETag`, with `304 Not Modified` on a matching `If-None-Match`. - A card is advisory: clients "MUST NOT treat Server Card contents as authoritative for security or access-control decisions" and should prefer live `server/discover` values when they disagree. ## Key Parameters | Artifact | Path / location | Status as the source states it | |---|---|---| | API catalog | `/.well-known/api-catalog` | RFC 9727, Internet Standards Track | | Link header | `Link: <...>; rel="api-catalog"` | RFC 8288, Standards Track | | AI Catalog | `/.well-known/ai-catalog.json` | Draft ("Protocol Revision: draft" in MCP docs) | | MCP Server Card | `/server-card` | See conflict below | | Cloudflare's example card | `/.well-known/mcp/server-card.json` | Cloudflare calls it "a proposal currently in draft" | | MCP discovery URI draft | `/.well-known/mcp-server`, `mcp://` scheme | draft-serra-mcp-discovery-uri-04, Intended status: Informational, dated 25 March 2026, expires 25 September 2026 | | Agent Skills index | `.well-known/agent-skills/index.json` | Cloudflare: "We've proposed" | | HTTP signature keys | `/.well-known/http-message-signatures-directory` | See [[concepts/agent-identity-web-bot-auth]] | | UCP profile | `/.well-known/ucp` | Part of Google's UCP; see [[concepts/agentic-commerce-protocols]] | ### Conflict: are Server Cards accepted? - The MCP site's SEP-2127 page shows status **Final**, type Extensions Track, created 2026-01-21, with a note that it is a historical record and the detailed wire format is maintained in the experimental repo. - The experimental repo's README says "Status: Experimental... not an accepted or official MCP extension" and that the SEP "is still under review". Its roadmap file says SEP-2127 "is in review" and that the Extensions Track also needs a merged SDK reference implementation. Both were fetched on the same day; the pages may have been updated at different times. The SEP page itself says Final status is "preserved as a historical record of the design as accepted" and defers to the current specification; no gathered source reconciles this with the repo's Experimental label. ### Conflict: where is the card? Cloudflare's Agent Readiness post (and isitagentready.com's checker) uses `/.well-known/mcp/server-card.json` with a `$schema` of `https://static.modelcontextprotocol.io/schemas/mcp-server-card/v1.json`, and a card shape that includes `serverInfo`, `transport` and a `tools` array. The MCP repo's discovery doc uses the AI Catalog plus `/server-card`, and the card schema omits tool listings. These are different shapes of the same idea; the MCP-hosted documents are the more authoritative source on the MCP side. ### What scanners check (isitagentready.com) The scanner lists: robots.txt, Sitemap, Link headers, DNS for AI Discovery (DNS-AID); Markdown negotiation; AI bot rules, Content Signals, Web Bot Auth; API Catalog, OAuth discovery, OAuth Protected Resource, Auth.md, MCP Server Card, A2A Agent Card, Agent Skills, WebMCP, ARD manifest, Personal Agent Protocol (Poppy); and commerce (x402, MPP, UCP, ACP). Cloudflare's launch post scores four dimensions and says commerce checks "do not currently count towards the score". Cloudflare sells bot-control products and runs the scanner; the checklist is its view of the ecosystem. ## When To Use - You operate a public API: publish the RFC 9727 catalog, the only dedicated catalog format here that is a finished RFC (RFC 8288 Link headers are also Standards Track). - You run a remote MCP server and want clients and registries to find it without manual configuration: an AI Catalog entry pointing at a Server Card, accepting that the format may change. - You already emit `Link` headers: adding the `api-catalog` relation costs little. ## Risks & Pitfalls - **Adoption is near zero.** Cloudflare Radar scanned the 200,000 most visited domains (after filtering categories such as redirects and ad servers) and reported, as of its launch post: `robots.txt` on 78% of sites, Content Signals on 4%, `Accept: text/markdown` handling on 3.9%, and MCP Server Cards plus RFC 9727 API Catalogs "together on fewer than 15 sites". This is a vendor-run scan of one population, not agent behaviour; it says nothing about whether any agent reads these files. - **No evidence of agent use in the gathered sources.** Wiki analysis: the sources document publication and scanner support, not fetch rates by agents. - **Churn.** The card path, schema URL and shape differ across sources; the AI Catalog and Server Card specs are drafts; the discovery URI draft expires 25 September 2026 (before 2026-11-04; no later revision is in the gathered sources). - **Information disclosure.** Public catalogs must not contain credentials, internal topology or proprietary logic (MCP discovery doc). A card can also mislead a client; the MCP docs describe stale or crafted cards as "a mild confusion or downgrade vector". - **Scanner scores are not standards compliance.** The checker mixes RFCs, drafts and one vendor's own proposals. ## Related Concepts - [[concepts/webmcp]] — in-page tools, which agents can only find by visiting the page. - [[concepts/agent-identity-web-bot-auth]] - [[concepts/llms-txt]] — human/LLM-readable site summary, a different kind of discovery. - [[entities/cloudflare]] ## Sources - `raw/github_doc-readme-md-10.md` — MCP Server Card extension: experimental status, scope, `$schema` convention. - `raw/github_doc-docs-discovery-md.md` — AI Catalog path and entry fields, card location, headers, security. - `raw/github_doc-docs-core-maintainer-roadmap-md.md` — SEP-2127 "in review" and the roadmap themes. - `raw/web_community-sep-2127-mcp-server-cards-http-server-discovery-model-contex.md` — SEP page showing Final status. - `raw/web_community-rfc-9727-api-catalog-a-well-known-uri-and-link-relation-to-h.md` — RFC 9727 requirements. - `raw/web_community-rfc-8288-web-linking-rfc-editor.md` — RFC 8288 status and Link header examples. - `raw/web_community-introducing-the-agent-readiness-score-check-to-see-if-your-s.md` — Cloudflare scanner, Radar adoption numbers, example paths. - `raw/web_community-is-your-site-agent-ready.md` — the scanner's current check list. - `raw/web_community-draft-serra-mcp-discovery-uri-04.md` — `/.well-known/mcp-server` and `mcp://` draft. - `raw/github_doc-docs-documentation-core-concepts-md.md` — UCP profile at `/.well-known/ucp`.