Put every answer in the first HTML

• Find: a 403 prevents that fetch. Block only on purpose.
• Understand: publish prices, limits, and requirements as HTML text. Link the docs from the homepage.
• Use: publish the next-step instructions relevant to your product. For an API, MCP server, or CLI, link authentication instructions, the public contract or entry point, and request examples from your docs.
• Test: inspect the HTTP status and saved HTML with the curl check below. A literal text match checks this response, not real crawler access or comprehension.
• This is the how-to for agent readiness on published pages. HonestClaw does not sell scores, reviews, or placement.

1. Bot rules that catch agents. Firewalls tuned for scrapers often return 403 to agent user-agents or to the shared cloud IP ranges agents fetch from. In the controlled probes Vercel published in 2026 (190 fetch probes across 19 site configurations), a 403 stopped both the plain fetch client and the JavaScript-running client, five out of five times. It is also easy to do without meaning to: Cloudflare's July 1 announcement scheduled September 15 defaults that would block Training and Agent categories on ad-bearing pages for newly onboarded domains. Check your configured rules. If you block a path, block it on purpose and know which one.
2. A robots.txt written for search engines. In Cloudflare's April 17, 2026 Agent Readiness launch sample of the top 200,000 domains (after category exclusions), 78% have a robots.txt, and most of those are written for search crawlers, not agents. A robots.txt rule that disallows AI crawlers stops a compliant agent before it sees the answer. A public site does not need a robots.txt to permit crawling. Use it only for deliberate exclusions, and read it once a quarter.
3. No page that states the offer. Agents base their answers on pages they actually fetched. In the same run study (1,033 agent runs across 25 sites), 844 of 1,023 scorable runs, or 82.5%, traced an answer back to a fetched page, and docs pages were the largest single source. Publish a canonical public page stating what the product does, and link it from your homepage.

Keep a sitemap to help crawlers and search systems enumerate canonical pages. Agents fetched sitemap.xml directly in 43 of 1,033 runs, or about 4%.

The first HTML test. Fetch the page's crawlable HTML without running JavaScript. If the fact you want an agent to use is in that response, you pass. If the response is an empty page shell and a script tag, you fail, no matter how good the page looks in a browser. Vercel and MERJ observed no JavaScript execution by GPTBot or ClaudeBot in their December 2024 measurements; the same report noted that Googlebot, Gemini, and AppleBot did execute JavaScript. Vercel's 2026 controlled probes separately showed that a fetch-only client missed an answer hidden behind JavaScript.

Write the fact as a sentence. A page an agent can fetch but cannot make sense of still loses. Publish prices, limits, and requirements as HTML text, including answers otherwise confined to images, PDFs, sales decks, or support threads.

The homepage is a map. In the run study the homepage was reached in 69% of runs and was the first fetch in 92% of those. The most common next hop was a docs page, in 59% of the runs that reached the homepage. Docs pages were reached in 83% of all runs, and agents that got there read about three and a half of them. So the homepage has one job: say what the product is and link to the docs that answer how. A vague call to action, a menu that only exists in JavaScript, or a dead docs link can prevent clients from reaching the intended answer.

Link your discovery files. Publishing them is not enough. llms.txt, openapi.json, and .well-known paths only help if agents find them, and in the run study the dominant path was following links. 97% of openapi.json fetches, 93% of .well-known fetches, and 86% of llms.txt fetches arrived through a link. Link these files with ordinary HTML links in your footer, docs sidebar, or a resources page. Use rel="alternate" for an alternate version of the current page, such as its Markdown mirror, not as a blanket relation for discovery files. A file with no inbound link is easy to miss; links are the dominant path in the study, not the only one.

Offer Markdown, once the HTML works. Markdown is an optional efficiency once the first HTML already carries the facts. In the study harnesses (not a claim that every agent prefers Markdown), 65% of all web fetches requested Markdown, and when a fetch named any format, it named Markdown 96% of the time. Keep HTML as the source of truth, and preserve its facts and links in any Markdown alternate. Cloudflare reported 16,180 HTML tokens versus 3,150 Markdown tokens for its announcement post. That is one conversion example, not a universal saving. Serve Markdown through content negotiation on Accept: text/markdown, or through a linked .md alternate. In the controlled probes, adding Markdown mirrors, llms.txt, JSON-LD, or a sitemap did not change whether a client could retrieve an answer that was already in the server-rendered HTML.

Send honest status codes. Agents read status codes literally. A missing page returns 404, not a 200 with "not found" on it. Throttling returns 429 with Retry-After, not a silent challenge page. Serve Markdown with Content-Type: text/markdown. When the same URL selects HTML or Markdown using Accept, include Vary: Accept on both representations so caches keep them apart. To a machine, a soft 404 or a hidden rate limit looks like a real page.

JSON-LD is worth adding where it is accurate. These controlled probes measured answer retrieval, not improvements in agent comprehension from JSON-LD. Keep critical facts in visible text, because some converters drop script blocks.

Publish the next-step instructions relevant to your product so an agent that found and understood you can continue. For an informational site, that may be contact paths, docs, or purchase steps on published pages.

Where your product exposes an API, MCP server, or CLI, also publish:

1. Authentication instructions, credential prerequisites, scopes, and any required human approval.
2. The public contract or entry point, linked from the docs.
3. Request examples and documented responses and errors.

Use checks whether these published materials are present and parseable. ARS does not measure live API/MCP call success or whether an agent finishes a product task. OAuth metadata describes server endpoints and capabilities; its publication alone does not demonstrate unattended credential acquisition.

On the seven study sites that shipped an openapi.json, agents fetched it in about one run in five.

llms.txt: citation association and answer grounding. SE Ranking reported no citation association in its sample. Trakkr's top-5,000 comparison was not significant; its full 37,894-domain comparison was significant with a small reported effect size, r = −0.065. Neither establishes what adding the file would cause. Separately, the run study measured a different question: among runs that reached an llms.txt, 36% grounded the final answer in it. Being cited and being usable by an agent are different things, and getting cited vs agent-ready covers that gap.

Protocol adoption in the April 17 launch sample. Cloudflare's snapshot started with the top 200,000 domains and excluded specified categories (redirects, ad servers, and tunneling services). In the resulting sample, 3.9% supported Markdown negotiation, 4% declared AI usage preferences through Content Signals, and MCP server cards or API catalogs appeared on fewer than 15 sites. These measure adoption of particular standards, not overall readiness across the web.

Standards checks and answer checks. Standards checks can confirm publication and serving behavior. If that is your question, those checks are enough. Answer checks address whether a reader can recover specific facts from fetched content. Neither demonstrates completed product tasks. A file that claims something the site cannot do is worse for an agent than no file at all.

Run this against the page you most want agents to read, and pick a fact that should be on it:

curl -sSL -A 'GPTBot' \
  -H 'Accept: text/html' \
  -D agent-headers.txt \
  -o agent-page.html \
  -w 'Final HTTP status: %{http_code}\nFinal Content-Type: %{content_type}\n' \
  https://yoursite.com/pricing &&
grep -Fc -- 'the fact you expect' agent-page.html

Check the final status and Content-Type before interpreting the count. A 403 blocks this request. Only treat a 200 response served as text/html as an HTML check. A positive count means the exact phrase appears somewhere in the saved response, not necessarily in readable page text. Zero means no exact match; inspect markup, encoding, and whitespace before concluding that the fact is absent.

The user-agent string changes one request header. It does not reproduce a real crawler's identity or network origin. Check verified crawler requests in server logs to assess actual access. A text match does not establish comprehension. GPTBot is OpenAI's crawler; ClaudeBot, PerplexityBot, or any other published agent name works the same way for this diagnostic, and none of them need an account.

• Fetch checks show whether content was reachable in the first response. They do not measure whether an agent understood it or completed a task with it.
• These findings describe their samples. Six sites account for two thirds of the run study. Cloudflare's April 17 launch snapshot covers selected categories drawn from popular domains, not the long tail.
• A free scan on HonestClaw creates a public record from the published surfaces it could reach. It stays logged out, fetches without running site JavaScript, and honors robots.txt. Where Agent Comprehension is available, it shows informational answers and evidence from the pages the scan read. It does not change the Agent Readiness Score.

1. Run the 30-second check on your homepage, your pricing page, and your docs landing page.
2. Fix unintended 403 responses on pages you intend agents to fetch, then first-HTML omissions, incorrect status codes, and missing discovery links.
3. Add authentication, contract, and request documentation for interfaces your product exposes.

• Vercel, The rise of the AI crawler (Dec 2024, with MERJ): GPTBot/ClaudeBot no-JS; Googlebot, Gemini, and AppleBot JS render noted.
• Vercel, Make your site readable by AI agents (Aug 2026): the 1,033-run study (844 of 1,023 scorable runs with answer provenance) and the 190 controlled fetch probes.
• Cloudflare, Introducing the Agent Readiness score (Apr 17, 2026): the 200,000-domain launch-snapshot adoption figures (after category exclusions), including 78% robots.txt.
• Cloudflare, Introducing Markdown for Agents (Feb 12, 2026): announcement-post token example (16,180 HTML vs 3,150 Markdown).
• Cloudflare, Your site, your rules (Jul 1, 2026): scheduled September 15 defaults that would block Training and Agent categories on ad-bearing pages for newly onboarded domains.
• SE Ranking, llms.txt study (Nov 2025): about 300,000 domains, no observed citation association in its sample.
• Trakkr Research, The llms.txt Effect (Mar 2026): 37,894 cited domains; top-5,000 citation difference not significant; full-sample small effect size r = −0.065.