Find out which AI and search crawlers your robots.txt allows or blocks — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and 14 more — plus noindex and llms.txt checks. Free per site; Pro adds bulk checks, CSV and a robots.txt rule builder.
- Free: one site at a time, all 19 crawlers
- Pro: 20 sites per run + CSV export
- Pro: robots.txt rule builder (allow AI search, block training — or your own mix)
- One-time payment, private link
Good to know
- We apply the robots.txt rules the way crawlers do: the most specific user-agent group wins, then the longest matching rule; on a tie, Allow wins.
- Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot …) and search/answer crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot …) are different. Blocking training does not have to mean disappearing from AI answers.
- robots.txt is honoured voluntarily. Firewall and CDN bot settings are not visible from outside.
- Want the full kit? See the llms.txt & AI-Crawler Starter Kit or the llms.txt Generator.
Frequently asked questions
Which crawlers do you check?
Nineteen common AI and search crawlers, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot and others, plus noindex and llms.txt.
Is the single-site check free?
Yes. Pro (€7 one-time) adds 20 sites per run, CSV export and a robots.txt rule builder.
If robots.txt allows a bot, is my site definitely crawled?
Not necessarily. robots.txt is honoured voluntarily, and a firewall or CDN bot setting can still block crawlers. We cannot see those settings from outside.
Want ready-made rules and an llms.txt?
See the llms.txt & AI-Crawler Starter Kit or the AI-Ready Pack, linked below.
Related products
llms.txt & AI-Crawler Starter Kit (DIY)
Copy-paste templates to make your site easier for AI assistants to read: llms.txt, robots.txt rules for AI crawlers, and schema.org JSON-LD.
AI-Ready Pack
Custom llms.txt, AI-crawler robots.txt rules and schema.org JSON-LD, written for your website by hand.
llms.txt Generator
Enter your website and get a ready-to-upload llms.txt draft built from your sitemap, page titles and descriptions. Free for 8 pages; one-time Pro unlocks 40 pages.
Search Console "Not Indexed" Fix Guide
Every common "Why pages aren't indexed" reason in Google Search Console, explained in plain language — with whether to worry and the fix, in order.