wikis / Agentic SEO / wiki / concepts / agent-discovery-files.md view as markdown report a mistake

Agent Discovery Files

Definition

Discovery files are small machine-readable documents or HTTP headers a site publishes at predictable locations so an agent can find out what the site offers (APIs, MCP servers, skills, commerce endpoints) before it interacts. They answer "where do I connect?" the way robots.txt answers "what may I fetch?". This page covers the agent-capability layer; crawl rules and AI-usage preferences are in robots txt for ai and content signals and aipref, and site summaries for LLMs in llms txt.

Most items below are drafts or experiments. Status is given per item; do not read presence in a scanner as evidence of a standard.

How It Works

Agents try fixed paths (usually under /.well-known/, RFC 8615) or read an HTTP Link header, then follow URLs in what they find.

RFC 9727: api-catalog (Internet Standards Track)

• Well-known URI: /.well-known/api-catalog. A publisher must resolve an HTTPS GET to it with an API catalog document, and an HTTPS HEAD with a Link header carrying the relations the RFC defines.
• The document must be in Linkset format, application/linkset+json (RFC 9264), and should carry a profile parameter with the value https://www.rfc-editor.org/info/rfc9727.
• It also defines the api-catalog link relation, which a page or response can use to point at the catalog wherever it lives.
• It does not mandate where API endpoints live or what their documentation looks like; the catalog lists them and links to specs, docs and status endpoints.

RFC 8288: Web Linking (Internet Standards Track, October 2017)

The Link response header carries typed links in the HTTP response itself, so an agent can find resources without parsing HTML. Cloudflare's readiness post gives this example:

HTTP/1.1 200 OK
Link: </.well-known/api-catalog>; rel="api-catalog"

AI Catalog and MCP Server Cards (draft, with a status conflict)

• The MCP experimental extension repo defines an AI Catalog served at /.well-known/ai-catalog.json, ideally with media type application/ai-catalog+json. Clients doing domain-level discovery "SHOULD attempt to retrieve" it. The catalog format itself is maintained in a separate Agent-Card/ai-catalog repository referenced by the MCP docs.
• A catalog entry for an MCP server has identifier, type (must be application/mcp-server-card+json), and one of url or data. Identifiers for open systems use urn:air:{publisher}:{namespace}:{name}.
• A Server Card describes a remote MCP server: name (reverse-DNS, e.g. com.example/weather), version, description, transport endpoints. It intentionally omits tools, resources and prompts, which remain subject to runtime listing.
• Reserved default card location: MAY host at GET <streamable-http-url>/server-card; any unreserved URI is valid, and a catalog entry's url gives the exact location. The repo's "alternatives considered" section explicitly does not recommend a .well-known path for individual cards.
• Clients fetching a card SHOULD send Accept: application/mcp-server-card+json. Hosted cards MUST be served over HTTPS (TLS 1.2 or later) in production; HTTP MAY be used for local development only. Hosted card endpoints MUST include CORS headers (Access-Control-Allow-Origin: *, Access-Control-Allow-Methods: GET, Access-Control-Allow-Headers: Content-Type, If-None-Match, Access-Control-Expose-Headers: ETag). Caching headers are SHOULD: Cache-Control: public, max-age=3600 plus an ETag, with 304 Not Modified on a matching If-None-Match.
• A card is advisory: clients "MUST NOT treat Server Card contents as authoritative for security or access-control decisions" and should prefer live server/discover values when they disagree.

Key Parameters

Conflict: are Server Cards accepted?

• The MCP site's SEP-2127 page shows status Final, type Extensions Track, created 2026-01-21, with a note that it is a historical record and the detailed wire format is maintained in the experimental repo.
• The experimental repo's README says "Status: Experimental... not an accepted or official MCP extension" and that the SEP "is still under review". Its roadmap file says SEP-2127 "is in review" and that the Extensions Track also needs a merged SDK reference implementation.

Both were fetched on the same day; the pages may have been updated at different times. The SEP page itself says Final status is "preserved as a historical record of the design as accepted" and defers to the current specification; no gathered source reconciles this with the repo's Experimental label.

Conflict: where is the card?

Cloudflare's Agent Readiness post (and isitagentready.com's checker) uses /.well-known/mcp/server-card.json with a $schema of https://static.modelcontextprotocol.io/schemas/mcp-server-card/v1.json, and a card shape that includes serverInfo, transport and a tools array. The MCP repo's discovery doc uses the AI Catalog plus /server-card, and the card schema omits tool listings. These are different shapes of the same idea; the MCP-hosted documents are the more authoritative source on the MCP side.

What scanners check (isitagentready.com)

The scanner lists: robots.txt, Sitemap, Link headers, DNS for AI Discovery (DNS-AID); Markdown negotiation; AI bot rules, Content Signals, Web Bot Auth; API Catalog, OAuth discovery, OAuth Protected Resource, Auth.md, MCP Server Card, A2A Agent Card, Agent Skills, WebMCP, ARD manifest, Personal Agent Protocol (Poppy); and commerce (x402, MPP, UCP, ACP). Cloudflare's launch post scores four dimensions and says commerce checks "do not currently count towards the score". Cloudflare sells bot-control products and runs the scanner; the checklist is its view of the ecosystem.

When To Use

• You operate a public API: publish the RFC 9727 catalog, the only dedicated catalog format here that is a finished RFC (RFC 8288 Link headers are also Standards Track).
• You run a remote MCP server and want clients and registries to find it without manual configuration: an AI Catalog entry pointing at a Server Card, accepting that the format may change.
• You already emit Link headers: adding the api-catalog relation costs little.

Risks & Pitfalls

• Adoption is near zero. Cloudflare Radar scanned the 200,000 most visited domains (after filtering categories such as redirects and ad servers) and reported, as of its launch post: robots.txt on 78% of sites, Content Signals on 4%, Accept: text/markdown handling on 3.9%, and MCP Server Cards plus RFC 9727 API Catalogs "together on fewer than 15 sites". This is a vendor-run scan of one population, not agent behaviour; it says nothing about whether any agent reads these files.
• No evidence of agent use in the gathered sources. Wiki analysis: the sources document publication and scanner support, not fetch rates by agents.
• Churn. The card path, schema URL and shape differ across sources; the AI Catalog and Server Card specs are drafts; the discovery URI draft expires 25 September 2026 (before 2026-11-04; no later revision is in the gathered sources).
• Information disclosure. Public catalogs must not contain credentials, internal topology or proprietary logic (MCP discovery doc). A card can also mislead a client; the MCP docs describe stale or crafted cards as "a mild confusion or downgrade vector".
• Scanner scores are not standards compliance. The checker mixes RFCs, drafts and one vendor's own proposals.

Related Concepts

• webmcp — in-page tools, which agents can only find by visiting the page.
• agent identity web bot auth
• llms txt — human/LLM-readable site summary, a different kind of discovery.
• cloudflare

Sources

• raw/github_doc-readme-md-10.md — MCP Server Card extension: experimental status, scope, $schema convention.
• raw/github_doc-docs-discovery-md.md — AI Catalog path and entry fields, card location, headers, security.
• raw/github_doc-docs-core-maintainer-roadmap-md.md — SEP-2127 "in review" and the roadmap themes.
• raw/web_community-sep-2127-mcp-server-cards-http-server-discovery-model-contex.md — SEP page showing Final status.
• raw/web_community-rfc-9727-api-catalog-a-well-known-uri-and-link-relation-to-h.md — RFC 9727 requirements.
• raw/web_community-rfc-8288-web-linking-rfc-editor.md — RFC 8288 status and Link header examples.
• raw/web_community-introducing-the-agent-readiness-score-check-to-see-if-your-s.md — Cloudflare scanner, Radar adoption numbers, example paths.
• raw/web_community-is-your-site-agent-ready.md — the scanner's current check list.
• raw/web_community-draft-serra-mcp-discovery-uri-04.md — /.well-known/mcp-server and mcp:// draft.
• raw/github_doc-docs-documentation-core-concepts-md.md — UCP profile at /.well-known/ucp.