Яндекс.Метрика

Markdown for AI agents: .md page versions and Accept: text/markdown

About 80% of a page's HTML is menus, scripts and markup. A Markdown version of the same page is several times shorter, so the agent spends its context on meaning, not boilerplate.

Why it matters

  • Claude Code and Cursor ask for Markdown themselves with Accept: text/markdown. In Evil Martians' logs, Claude Code made 94% of Markdown requests to their site.
  • A Markdown version at the same URL is not cloaking: same content, different format, and Vary: Accept tells caches so.
  • In our score, Markdown via Accept and .md page versions together weigh 12 out of 100 — more than llms.txt.

What to do

  1. 1

    .md page versions

    Every page gets a twin ending in .md: /prices → /prices.md, the homepage → /index.md. Generate them from the same source as the HTML, or the texts will drift apart.

  2. 2

    Respond to Accept

    For a request with Accept: text/markdown, the same URL returns Markdown with Content-Type: text/markdown and Vary: Accept. Compare q-values instead of matching a substring.

    curl -sD - -o /dev/null -H "Accept: text/markdown" https://example.com/ | grep -iE "content-type|vary"
  3. 3

    Say where the Markdown is

    A link in <head> for DOM parsers and a Link header for HTTP clients:

    <link rel="alternate" type="text/markdown" href="/prices.md">
Link: </prices.md>; rel="alternate"; type="text/markdown"

What the scanner checks

  • The site points agents to its machine-readable files

    Links in the response header or page code lead to llms.txt, a Markdown version or an API description.

  • The site serves Markdown when an agent asks

    An agent can ask for a page in Markdown instead of HTML and spend far less of its budget on markup.

  • Pages have Markdown versions

    The same page at a URL ending in .md: clean text with no menus or scripts.

  • The Markdown version has a header with page data

    Title, description and URL at the top of the file. An agent learns what the document is without parsing the text.

  • Pricing is available in Markdown

    An agent comparing products reads pricing most easily from a plain text file.

Check your own site

Paste the address and get the same breakdown in half a minute.

Check a site →

Sources

Other guides