Free tool
AI crawler robots.txt checker
Paste your robots.txt to see which AI crawlers it allows and which it blocks, for OpenAI, Anthropic, Perplexity and Google, and the exact line that decided each one. It is free, needs no sign-up, and runs entirely in your browser.
Check a robots.txt
Open yoursite.com/robots.txt, copy everything it shows, and paste it below.
How do I read the result?
Each row is one crawler: whether your file lets it fetch the page, and which line of the file decided that.
The crawlers do three jobs. Search crawlers index your pages so an assistant can cite them. Training crawlers collect content that may be used to train models. Crawlers marked For a person fetch one page because someone asked about it. If you want to be named in AI answers, the Search rows are the ones that must say Allowed. The names and jobs come from each company's own documentation; see AI crawler user agents.
A crawler follows the group that names it and ignores every other group. So a file with a group for GPTBot and a general group marked with a star gives GPTBot only the rules in its own group. Within a group the longest matching path wins, and Allow wins a tie.
What can this tool not check?
Anything outside the file: your firewall, your CDN's bot settings, CAPTCHAs, and whether your pages show their text without JavaScript.
robots.txt is a request that well-behaved crawlers follow. A security setting in front of your site can block a crawler whatever the file says, and a page that builds its text with scripts can look empty to a crawler that is allowed in. The other four checks are in how to check if AI crawlers can access your website. The free Kortex audit checks crawler access from outside and shows what ChatGPT, Gemini and Claude say about your company.
How do I fix a block?
Replace the rule that the result points to, then check the file that your site actually serves.
We keep three files you can copy, for allowing everything, for allowing AI search while opting out of training, and for blocking all AI use, in robots.txt for AI crawlers. If you have not decided which you want, read should you block AI crawlers? OpenAI says its search systems take about 24 hours to adjust after a robots.txt change.
Questions
Frequently asked questions
Is this AI crawler checker free?
Yes. It runs in your browser, needs no sign-up and sends nothing anywhere.
Can it fetch my robots.txt for me?
No. Browsers stop one site from reading another site's files. Open yoursite.com/robots.txt, copy what it shows and paste it in.
Does a green result mean AI crawlers can read my site?
It means your robots.txt allows them. A firewall, a CDN bot setting or a CAPTCHA can still block a crawler, and this tool cannot see those.
Why does a crawler ignore my User-agent: * rules?
A crawler follows the group that names it and ignores the rest. If your file has a group for GPTBot, GPTBot reads only that group, not the * group.
Which AI crawlers should I allow?
To be named in AI answers, allow the search crawlers: OAI-SearchBot, Claude-SearchBot and PerplexityBot. Blocking the training crawlers is a separate choice.
Read next
- llms.txt generator and checkerWrite an llms.txt for your site, or check one against the format. Free, in your browser.
- AI visibility auditCheck if ChatGPT, Gemini, Claude and Google name your company for the questions your customers ask, who they name instead, and get the fixes written for free.
- GlossaryAI visibility, AI Overviews, llms.txt, ROAS, MER and more, in plain words.
See whether AI recommends you.
20 questions your customers ask, put to ChatGPT, Gemini and Claude, plus Google, and four fixes written for you. Free, one per website.