Яндекс.Метрика

robots.txt for AI: YandexAdditional, GPTBot, ClaudeBot and Content-Signal

Almost every site has a robots.txt, but it is written for search engines. AI bots do different jobs: some collect training data, others fetch an answer for a user right now. Tell them apart and list them explicitly.

Why it matters

  • Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) collect data for models. Blocking them is a legitimate choice, and our score does not drop for it.
  • AI search robots and user agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) come when someone asks about you. Block them and you disappear from answers.
  • Yandex controls whether a site appears in Alice AI answers through the YandexAdditional and YandexAdditionalBot robots. Without a separate group, the general * rule applies.
  • Sber does not publish the name of the robot that collects data for GigaChat, so there is no way to address a rule to it yet.

What to do

  1. 1

    List AI bots explicitly

    Separate groups tell agents the policy is deliberate. An example for a site that wants to appear in AI answers:

    User-agent: YandexAdditional
User-agent: YandexAdditionalBot
Allow: /

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# training is up to you
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Allow: /
  2. 2

    Add Content-Signal

    The Content-Signal line (contentsignals.org) says what content may be used for: search, AI answers, training. robots.txt validators flag it as unknown, which is harmless.

    User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
  3. 3

    Check the result

    Open robots.txt without a browser and find your groups.

    curl -s https://example.com/robots.txt | grep -iE "user-agent|content-signal"

What the scanner checks

  • robots.txt exists

    It is the first file any robot reads: the rules for what it may open on the site.

  • robots.txt has rules for AI bots

    AI bots are listed by name. Whether you allow or block them, it shows the choice was made.

  • Content use is declared

    The Content-Signal line says whether your text may be used for search, AI answers and model training.

  • Alice AI robots are allowed

    The YandexAdditional robots bring a site into Alice AI answers. Best to allow them with a rule of their own.

  • Agents and AI search are allowed

    ChatGPT, Claude and Perplexity come when someone asks about you. Block them and you vanish from answers.

  • The page does not forbid quoting its text

    The nosnippet and noai tags forbid showing page text in answers. With them an assistant stays silent about you or gives only a link.

  • The site does not forbid using its text in AI answers

    A Content-Signal line can forbid search and AI answers. Forbidding model training is your call and does not affect the result.

  • Pricing and docs are not closed to agents

    The homepage may be open while the pricing or docs section is blocked in robots.txt. Then an agent cannot answer a pricing question or set up an integration.

Check your own site

Paste the address and get the same breakdown in half a minute.

Check a site →

Sources

Other guides