AI Crawler Access Control

This parameter checks whether your robots.txt blocks the named AI crawler tokens that feed engines like ChatGPT, Claude, Perplexity, and Gemini. friendly4AI checks each tracked bot on its own and scores on the ratio that come back blocked. Zero blocked passes (100). Block 75% or more and you fail (0). Everything in between lands as partial.

What does this parameter measure?

friendly4AI reads your robots.txt one bot at a time. It fetches the file, then works out the access decision for every tracked AI user-agent: GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI), ClaudeBot, Claude-Web (Anthropic), PerplexityBot, Perplexity-User (Perplexity), Google-Extended (Google), plus CCBot, Bytespider, Amazonbot, and Meta-ExternalAgent. The result is a simple count: how many of those bots are blocked, and how many are allowed.

Don't confuse this with robots.txt Accessibility, which tests the wildcard (User-agent: *) rule for the whole site. This check ignores the wildcard and looks only at the per-AI-crawler tokens.

Why does AI crawler access matter for AI-readiness?

A site can sail through a generic robots.txt check and still shut out the bots that decide AI visibility. Every blocked token cuts off a specific engine:

• Block GPTBot and you drop out of ChatGPT's browsing and citations.
• Block OAI-SearchBot and you disappear from ChatGPT Search results.
• Block PerplexityBot and you vanish from Perplexity's answers.
• Disallow Google-Extended and your content opts out of Gemini and Vertex AI generative features.

These blocks tend to creep in by accident: a copied robots.txt snippet here, a CDN bot rule there. The rest of the site looks healthy while one AI engine after another quietly loses access. The check works alongside AI opt-out signals and AI usage policy.

How is AI Crawler Access Control scored?

Under the v4.5 methodology, this Crawlability parameter scores on the ratio of blocked AI crawlers to total tracked.

• Absent or empty robots.txt: nothing is restricted, so the parameter passes (100).
• Present robots.txt: friendly4AI resolves each tracked bot's access using longest-match Allow/Disallow precedence, then computes the blocked ratio:
  • 0% blocked → pass (100)
  • under 25% blocked → partial (75)
  • 25% to under 75% blocked → partial (50)
  • 75% or more blocked → fail (0)

If an operator configures a bot taxonomy, the scan switches to a tier-weighted mode. That mode gives high-value citation bots more weight and passes at a score of 50 or above. Without a taxonomy, the default flat ratio above applies.

How do I fix AI crawler access issues?

• Write out the exact tokens each engine uses (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended) and confirm none of them carry an unintended Disallow: /.
• Want AI access? Skip per-bot Disallow: / rules. An empty Disallow: or an explicit Allow: / keeps the bot allowed.
• Watch for a more specific bot group overriding a permissive wildcard. robots.txt precedence picks the longest-matching rule, so one stray bot block wins out.
• Keep CDN or WAF bot rules in sync with robots.txt, otherwise a blocked token gets re-introduced at the edge.
• Re-scan after you edit robots.txt to confirm the blocked ratio dropped and the tracked bots are allowed.