Skip to content

Lab

Can AI crawlers see your website?

Enter a URL and this checks it the way GPTBot, ClaudeBot, PerplexityBot and the rest would. It looks at whether they are allowed in, whether the server lets them through, and whether there is any text for them to read once they arrive.

A domain or a specific page.

What gets checked

  • robots.txt, evaluated per crawler for the exact URL, with the same group and longest-match rules the crawlers use. A bot with its own group ignores the * group entirely, which trips up a lot of sites.
  • Firewall and CDN blocks. The page is requested as a browser, then again as four AI crawlers. A refusal, a challenge page or much thinner content for the bots shows up here, even though robots.txt says nothing about it.
  • noindex and nosnippet in meta tags and the X-Robots-Tag header, including rules aimed at a single bot.
  • Content without JavaScript. Most AI crawlers read the HTML and never run scripts. A client-rendered page can look perfect in a browser and be empty to them. The rendering test logs which ones do run it.
  • Supporting signals: title, description, H1, structured data, canonical and sitemap.

Training crawlers are scored lightly. Opting out of model training is a legitimate choice, and it does not keep a site out of AI search answers; blocking the search and user-triggered fetchers does. Nothing you check here is stored.