robots.txt Accessibility

This parameter checks whether your robots.txt lets major AI crawlers fetch your site. friendly4AI passes a domain when there is no blanket Disallow: / under User-agent: * (or when robots.txt is absent and no blocking meta robots or X-Robots-Tag directive is present). It fails the domain when crawling or indexing is hard-blocked. This is a Crawlability parameter, stable since v4.2.

What this parameter checks

friendly4AI reads your robots.txt and looks at the rules that apply to the wildcard (User-agent: *) and to known AI crawler tokens: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and Google-Extended. When robots.txt is missing, the scan falls back to your meta robots and X-Robots-Tag directives. See also AI crawler access control and sitemap availability.

Why robots.txt matters for AI-readiness

AI systems read robots.txt before they fetch anything. Block GPTBot and ChatGPT can no longer browse or cite your site. Block PerplexityBot and you drop out of Perplexity's answers. Disallow Google-Extended and your content is opted out of Google's Gemini and Vertex AI generative features. Of every misconfiguration we measure, a blanket block does the most damage to AI visibility.

How robots.txt accessibility is scored

Scoring runs in two tiers.

• robots.txt present — Pass, unless a blanket Disallow: / under User-agent: * blocks crawling. That blanket block is a Fail. This tier has no partial.
• robots.txt absent — the scan falls back to meta robots and X-Robots-Tag. noindex, none, or nofollow is a Fail. A weak negative such as noarchive on its own is a Partial. Anything else is a Pass.

An unreachable robots.txt does not fail automatically. It routes to the fallback tier above.

How to fix robots.txt accessibility issues

• Don't add Disallow: / under User-agent: *, or under a specific AI crawler token like GPTBot, ClaudeBot, PerplexityBot, or Google-Extended, unless you actually mean to block it.
• Check your file's syntax against Google Search Central's robots.txt guide. One stray rule can block far more than you intended.
• Confirm the exact crawler tokens each engine uses (OpenAI's GPTBot documentation lists them), then allow the ones you want reaching you.
• If you depend on the meta robots fallback, keep noindex, none, and nofollow off any page you want surfaced.
• Re-scan after every change to confirm crawlers are still allowed.