# An agent can open your site: no CAPTCHA, no JavaScript wall, an honest 404

> Why an AI agent can't read a website: bot walls, CAPTCHAs, JavaScript-only content, soft 404s. How to check your site with one curl command and what to fix.

Источник: https://pachca.com/en/agent-ready/guide/site-access

An agent is an HTTP client without a browser. If your site answers with a CAPTCHA, an empty page before JavaScript runs, or the homepage instead of a 404, the agent learns nothing about the product and picks a vendor it could read.

## Why it matters

- ChatGPT, Claude, Perplexity and coding agents read the HTML the server returns. Most of them do not run JavaScript.
- DDoS protection (Qrator, DDoS-Guard, Cloudflare) often shows agents a browser check page. A person never sees it, but for the agent it is a dead end.
- A soft 404 — the homepage or a 200 for a missing URL — misleads the agent: it thinks the page exists and quotes the wrong thing.

## What to do

1.  1

    ### Check what an agent sees

    Request the homepage without a browser. You want a 200 and text about the product, not “enable JavaScript” or “checking your browser”.

    ```
    curl -sL -A "agent-readiness-index/0.1" https://example.com/ | sed -e 's/<[^>]*>//g' | wc -w
    ```

2.  2

    ### Render text on the server

    The homepage, pricing and product pages should be server-rendered or statically built. Prices belong in the HTML, not in a script that fills them in later.

3.  3

    ### Let agents through bot protection

    In Qrator, DDoS-Guard or Cloudflare settings, allow known agents and AI search crawlers. A CAPTCHA is the same as a closed site for them.

4.  4

    ### An honest 404

    A missing URL should return 404, not 200 and not a redirect to the homepage.

    ```
    curl -s -o /dev/null -w "%{http_code}\n" https://example.com/no-such-page
    ```

## What the scanner checks

- Homepage opens without a CAPTCHA

    An agent is a program without a browser. If the site shows it a CAPTCHA or a “checking your browser” page, it reads nothing.

- Text is visible without JavaScript

    Most agents do not run JavaScript. Product text has to be in the page itself, not loaded by a script.

- Prices are visible without JavaScript

    Agents are often asked to compare pricing. If a script fills in the prices, the agent never sees them.

- A missing page returns 404

    If the site answers “OK” to any URL, the agent takes an empty page for a real one.

- A short path from http to https

    A long redirect chain means extra requests and a chance the agent gives up.

- The homepage is light and fast

    A page over a megabyte or slower than three seconds eats the time and text budget an agent has.

- A clear heading structure

    One main heading, subheadings in order and a marked main area. An agent uses them to split the page into parts.

- The homepage is not hidden from search

    Agents often find a site through search. A noindex tag removes the page from results, so the agent never gets there.

- Homepage text fits an agent’s memory

    An agent can only read so much text at once. It will cut a page that is too long and lose the end.

- Content starts near the top of the page

    If a long menu comes before the main text, the agent spends its budget on it and may never reach the point.

- ChatGPT and Claude get the same page people do

    We open the homepage the way the ChatGPT and Claude agents do. If site protection answers them with a refusal or a CAPTCHA, a person never notices, and the agent leaves with nothing.

- AI search gets the same page people do

    We open the homepage the way the ChatGPT and Perplexity search robots do. If site protection turns them away, the site is missing from the search an agent uses to find you.

- Images have text descriptions

    An agent does not see images. It learns what a picture shows only from its text description.

Check your own site

Paste the address and get the same breakdown in half a minute.

[Check a site →](https://pachca.com/en/agent-ready)

## Sources

- [Evil Martians: Making your site visible to LLMs and AI agents](https://evilmartians.com/chronicles/how-to-make-your-website-visible-to-llms)
- [Ora methodology (Content without JavaScript, bot detection)](https://ora.ai/methodology)

## Other guides

- [robots.txt for AI: YandexAdditional, GPTBot, ClaudeBot and Content-Signal](https://pachca.com/en/agent-ready/guide/robots-txt)
- [llms.txt and llms-full.txt: what they are and whether you need them](https://pachca.com/en/agent-ready/guide/llms-txt)
- [Markdown for AI agents: .md page versions and Accept: text/markdown](https://pachca.com/en/agent-ready/guide/markdown)
- [sitemap, metadata and JSON-LD: how an agent learns who you are](https://pachca.com/en/agent-ready/guide/structured-data)
- [API and MCP for agents: OpenAPI, MCP Server Card, OAuth discovery](https://pachca.com/en/agent-ready/guide/api-mcp)