AI crawler checker
Paste a page from your site to see which AI crawlers your robots.txt lets in, whether a firewall turns them away, and whether the page has text and structured data they can read without running JavaScript. It's free and needs no sign-up.
Search, training and user crawlers are different
Search crawlers decide whether you're cited
OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot, Googlebot and Bingbot build the indexes AI answers draw on. If they're blocked, your pages can't be found or cited.
Training crawlers are a separate choice
GPTBot, ClaudeBot, Google-Extended and CCBot collect content for training models. OpenAI, Anthropic and Google run them separately from their search crawlers, and Google says Google-Extended doesn't affect Search, so you can block training and still be cited.
User-requested fetches work differently
ChatGPT-User, Claude-User and Perplexity-User visit a page when a person asks about it. Because a person requested it, their makers say robots.txt may not apply.
What this checker looks at
robots.txt rules for each crawler
We read your robots.txt the way the crawlers do (RFC 9309): a group that names the crawler applies to it, otherwise the “*” group does, and the most specific rule wins.
Firewall and CDN blocks
We request the page twice more, identifying as OAI-SearchBot and ClaudeBot. If those requests are refused while a normal one works, a firewall may be blocking AI crawlers even when robots.txt allows them.
Text without JavaScript
AI crawlers read the HTML your server sends and don't run scripts. We count the words in that HTML, the way they see it.
Indexing, structured data, sitemap and llms.txt
We look for noindex rules, JSON-LD structured data, a sitemap and an llms.txt file: signals that help crawlers find the page and understand it.
Frequently asked questions
Common questions
How do I check if ChatGPT can crawl my website?
Paste a page into the checker above. It reads your robots.txt for OAI-SearchBot, which powers ChatGPT search, and GPTBot, which collects training data, and tests whether your firewall turns a request from OAI-SearchBot away.
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot only collects content for training. ChatGPT's search uses OAI-SearchBot, which you can allow separately. The same split exists at Anthropic (ClaudeBot and Claude-SearchBot) and Google (Google-Extended and Googlebot).
Why does my robots.txt allow a crawler but the checker shows a firewall warning?
Some CDNs and firewalls block AI crawlers by their user agent, regardless of robots.txt. Our test request uses the crawler's name but comes from our server, so a firewall that also checks IP addresses may treat it differently from the real crawler. Confirm in your CDN or firewall settings.
Do I need an llms.txt file?
It's optional. llms.txt is an emerging convention for giving AI tools a summary of your site, but no major AI company has said its crawlers rely on it. robots.txt, readable HTML and structured data matter more.
Is this checker free?
Yes, with no sign-up. We fetch your robots.txt, the page and a few public files once per check and don't store the results.
More guides
How to get cited by ChatGPT, Perplexity and other AI search engines
Verifiable claims, structured data and clear authorship in every article.
Read the guide→Does Google penalize AI-generated content?
What Google actually punishes, and how to publish with AI without exposing yourself.
Read the guide→How do you use AI for news without taking other outlets' reporting?
Credit and link the original story, start from your own material, and exclude outlets you don't want to use.
Read the guide→