# robots.txt for AI: YandexAdditional, GPTBot, ClaudeBot and Content-Signal

> How to set up robots.txt for AI: Alice AI robots (YandexAdditional), GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and the Content-Signal line. Example rules.

Источник: https://pachca.com/en/agent-ready/guide/robots-txt

Almost every site has a robots.txt, but it is written for search engines. AI bots do different jobs: some collect training data, others fetch an answer for a user right now. Tell them apart and list them explicitly.

## Why it matters

- Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) collect data for models. Blocking them is a legitimate choice, and our score does not drop for it.
- AI search robots and user agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) come when someone asks about you. Block them and you disappear from answers.
- Yandex controls whether a site appears in Alice AI answers through the YandexAdditional and YandexAdditionalBot robots. Without a separate group, the general `*` rule applies.
- Sber does not publish the name of the robot that collects data for GigaChat, so there is no way to address a rule to it yet.

## What to do

1.  1

    ### List AI bots explicitly

    Separate groups tell agents the policy is deliberate. An example for a site that wants to appear in AI answers:

    ```
    User-agent: YandexAdditional
    User-agent: YandexAdditionalBot
    Allow: /

    User-agent: OAI-SearchBot
    User-agent: ChatGPT-User
    User-agent: Claude-SearchBot
    User-agent: Claude-User
    User-agent: PerplexityBot
    User-agent: Perplexity-User
    Allow: /

    # training is up to you
    User-agent: GPTBot
    User-agent: ClaudeBot
    User-agent: Google-Extended
    Allow: /
    ```

2.  2

    ### Add Content-Signal

    The Content-Signal line (contentsignals.org) says what content may be used for: search, AI answers, training. robots.txt validators flag it as unknown, which is harmless.

    ```
    User-agent: *
    Content-Signal: search=yes, ai-input=yes, ai-train=yes
    Allow: /
    ```

3.  3

    ### Check the result

    Open robots.txt without a browser and find your groups.

    ```
    curl -s https://example.com/robots.txt | grep -iE "user-agent|content-signal"
    ```

## What the scanner checks

- robots.txt exists

    It is the first file any robot reads: the rules for what it may open on the site.

- robots.txt has rules for AI bots

    AI bots are listed by name. Whether you allow or block them, it shows the choice was made.

- Content use is declared

    The Content-Signal line says whether your text may be used for search, AI answers and model training.

- Alice AI robots are allowed

    The YandexAdditional robots bring a site into Alice AI answers. Best to allow them with a rule of their own.

- Agents and AI search are allowed

    ChatGPT, Claude and Perplexity come when someone asks about you. Block them and you vanish from answers.

- The page does not forbid quoting its text

    The nosnippet and noai tags forbid showing page text in answers. With them an assistant stays silent about you or gives only a link.

- The site does not forbid using its text in AI answers

    A Content-Signal line can forbid search and AI answers. Forbidding model training is your call and does not affect the result.

- Pricing and docs are not closed to agents

    The homepage may be open while the pricing or docs section is blocked in robots.txt. Then an agent cannot answer a pricing question or set up an integration.

Check your own site

Paste the address and get the same breakdown in half a minute.

[Check a site →](https://pachca.com/en/agent-ready)

## Sources

- [Yandex Webmaster: blocking display in Alice AI answers (in Russian)](https://yandex.ru/support/webmaster/ru/adding-site/indexing-prohibition)
- [OpenAI: crawlers and user agents](https://platform.openai.com/docs/bots)
- [Content Signals](https://contentsignals.org/)

## Other guides

- [An agent can open your site: no CAPTCHA, no JavaScript wall, an honest 404](https://pachca.com/en/agent-ready/guide/site-access)
- [llms.txt and llms-full.txt: what they are and whether you need them](https://pachca.com/en/agent-ready/guide/llms-txt)
- [Markdown for AI agents: .md page versions and Accept: text/markdown](https://pachca.com/en/agent-ready/guide/markdown)
- [sitemap, metadata and JSON-LD: how an agent learns who you are](https://pachca.com/en/agent-ready/guide/structured-data)
- [API and MCP for agents: OpenAPI, MCP Server Card, OAuth discovery](https://pachca.com/en/agent-ready/guide/api-mcp)