Check AI Crawler Access in robots.txt: GPTBot and Others

To check AI crawler access, fetch your robots.txt, find the rule group that applies to each AI user agent such as GPTBot, ClaudeBot or PerplexityBot, and see whether it disallows your pages. Do it for every agent you care about, and treat an unreadable file as unknown rather than allowed.

What robots.txt can and cannot do

robots.txt is a plain-text file at the root of a host that tells crawlers which paths they should not fetch. The syntax is defined in RFC 9309. It is a request, not a lock: compliant crawlers honour it, others may not, and it never hides a page from someone with the URL. If you need real access control, use authentication or server-level rules.

For AI, the file carries extra weight because it is the main published lever for expressing a policy toward model and answer-engine crawlers. Vendors document their user agent names, for example OpenAI on its bots page and Google in its crawler overview.

Agents worth checking

BusinessMCP’s AI crawler checker names these examples, and its API endpoint reports which of them robots.txt allows:

User agent	Operator or purpose	Notes
GPTBot	OpenAI	Documented by OpenAI
ClaudeBot	Anthropic	Named in BusinessMCP’s checks
PerplexityBot	Perplexity	Named in BusinessMCP’s checks
Google-Extended	Google	A robots.txt token for controlling certain AI uses of content
CCBot	Common Crawl	Open web crawl widely used in datasets

Lists change quickly. Treat any table, including this one, as a starting point and confirm names on each operator’s own documentation.

Key takeaway: write your policy per user agent, deliberately. A single User-agent: * block can unintentionally decide all of them at once.

How the matching actually works

Mistakes cluster around group selection.

1. A crawler uses the most specific User-agent group that matches its name. If a GPTBot group exists, the * group is ignored for that crawler.
2. Within a group, the longest matching path rule wins; Allow and Disallow conflicts follow the standard’s precedence rules.
3. Disallow: with no value allows everything. Disallow: / blocks everything.
4. Groups are separated by blank lines, and a stray blank line can split a rule from its user agent.

robots.txt snippet	Effect on GPTBot
User-agent: * / Disallow: /private/	Allowed except /private/
User-agent: GPTBot / Disallow: /	Fully blocked
User-agent: * / Disallow: / plus User-agent: GPTBot / Allow: /	Allowed; the specific group wins
No robots.txt (404)	Generally treated as no restrictions

Missing, forbidden and erroring files are treated differently by the standard and by individual crawlers, which is why a checker should not collapse them. BusinessMCP’s documentation says an unreadable robots.txt returns “unknown”, not “allowed”, a more cautious reading than many home-made scripts.

Common mistakes

• Leaving a staging Disallow: / in production. The classic. It blocks everything, including AI crawlers, and quietly tanks the AI readiness score, where crawler access carries 30 of 100 points.
• Blocking by CDN or firewall instead of robots.txt. The file says allowed, the edge returns 403. A robots.txt check will pass while crawlers fail. Compare with server logs; BusinessMCP has a guide on tracking AI crawlers server-side for that gap.
• Assuming blocked means unseen. A disallowed URL can still be known from links.
• Copying a list from a blog post. Agent names change; recheck periodically.

Doing it by hand

1. Open https://yourdomain.com/robots.txt and confirm HTTP 200.
2. For each agent, find its group, or the * group.
3. Test a few real URL paths against the rules, not just the homepage.
4. Repeat for subdomains, because each host has its own file.
5. Record the date; policies drift.

Automating it

By hand works for one site. For a portfolio, call an endpoint. BusinessMCP’s website checks API offers an AI crawler access check at $0.01 per call that reports which crawlers robots.txt allows. Run it weekly, store the result and alert on any change; our piece on scheduled website monitoring shows a pattern. A related file, llms.txt, is checked separately because it does not control access.

Deciding whether to allow or block AI crawlers is a business and legal question that depends on your content and goals. This article explains the mechanics only. Confirm current crawler names and policies with each operator.

A policy template to start from

Decide your position on three kinds of crawler, then write rules to match. The categories below are a way of organising the decision, not a recommendation.

Crawler type	Typical purpose	Questions to ask
Search or answer crawlers	Fetch pages to show in AI answers with links	Do I want visibility in AI answers?
Training crawlers	Collect content for model training	Am I comfortable with my content used this way?
User-triggered agents	Fetch a page because a person asked an assistant	Should an assistant be able to read this for my visitors?

Operators sometimes use separate user agents for these purposes and document them, so check each vendor’s page for the current names and what each controls. Google-Extended, for example, is a token that controls certain AI uses of content rather than a separate crawler that fetches pages.

Testing rules before you deploy them

1. Write the new robots.txt in a staging location.
2. Test sample URLs against it using a robots.txt tester or a library that implements the standard.
3. Confirm the file is served as text/plain with status 200.
4. Deploy, then fetch the live file and compare with the staged copy.
5. Run the crawler access check the next day to confirm the verdicts match your intent.

What to look for in server logs

robots.txt states intent, but logs show behaviour. Search access logs for the user agent strings of the crawlers you care about, and compare which paths they request with what your rules allow. A crawler that fetches disallowed paths is ignoring the file, and one that never appears may be blocked upstream or simply not interested. This is the gap BusinessMCP’s server-side tracking guide addresses. Do not rely on user agent strings alone, since they can be spoofed; vendors that publish IP ranges let you verify genuine traffic.

Keeping the policy current

Treat the file as living configuration. Put it in version control, review it when a new crawler gains attention and rerun your check after any change to your CDN, hosting or CMS. Platforms sometimes ship default robots.txt files or let plugins edit them, so a deploy you did not think was related can change your policy.