Google Ads budget calculator
Estimate the monthly budget, cost per lead and customers you can expect for your industry.
Open toolFree tool
AI crawler access checker
Check your robots.txt, llms.txt and page headers to see which AI search and training crawlers can read your site, from OpenAI, Anthropic, Perplexity, Google, Apple, Meta and others.
- Free, no sign-up
- Results in seconds
- Built by the JuvioX team
Want an expert to look at your results?
Send them with a free audit request, or book a free call and walk through them with a specialist.
Quick answer
How do I check if AI crawlers can access my website?
Look at your robots.txt file for rules that block AI crawlers such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended. This checker reads your robots.txt and shows which AI search and training crawlers are allowed or blocked, so you can decide what to change.
How it works
Three steps, no sign-up
-
01
Enter your website
Your home page or any page you want AI tools to be able to read.
-
02
We read your rules
We check robots.txt for each crawler on that exact page, plus llms.txt and noindex or noai tags.
-
03
Decide who gets in
Keep AI search crawlers allowed so you can be cited. Blocking training crawlers is your choice.
What is the difference between AI search and AI training crawlers?
Search and user crawlers, such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot, fetch pages to answer questions and cite sources. Training crawlers, such as GPTBot, ClaudeBot and CCBot, collect content to train models. Blocking search crawlers can stop you being cited; blocking training crawlers usually does not.
Does blocking Google-Extended remove me from Google?
No. Google-Extended only controls whether Google may use your content for Gemini models. Google Search, including AI Overviews, uses the normal Googlebot.
Do all AI crawlers respect robots.txt?
The major providers say their crawlers follow robots.txt, but it is a request, not a lock. To truly block a crawler you need firewall or server rules.
Should I block AI crawlers?
It depends on your goals. Blocking AI search crawlers can stop your content from appearing in AI answers, while blocking training crawlers only stops it from being used to train models. Most businesses that want AI visibility allow the search crawlers.
How do I allow or block an AI crawler?
Add rules to your robots.txt using the crawler's user agent name, for example "User-agent: GPTBot" followed by "Allow: /" or "Disallow: /". The change applies the next time the crawler checks your site.
More free tools
Keep going
Conversion tracking checker
Scan any page for GA4, Google Tag Manager, Google Ads, Meta Pixel and Consent Mode.
Open toolUTM link builder
Build clean, consistent UTM links for Google, Meta, email and social in seconds.
Open toolFree growth audit
Ready to grow with clarity?
Tell us about your goals. We'll reply within one business day with where you're losing money and what to fix first.
- Campaign and funnel review
- Tracking and data check
- Prioritised next steps
No obligation. No spam. Just clear next steps.