Training blocked · Search ✓ · User ✓ · Ads ✓
- GPTBotTrainingBlockedNot probed
- OAI-SearchBotSearchAllowedNot probed
- ChatGPT-UserUser fetchAllowedNot probed
- OAI-AdsBotAdsAllowedNot probed
What each operator's bots may fetch (robots.txt), what our synthetic fetch saw, and which answers can quote you.
Training blocked · Search ✓ · User ✓ · Ads ✓
Training ✓ · Search ✓ · User ✓
Search ✓ · Google-Extended ✓ · Agent ✓
Search blocked · User ✓
Search ✓
Search ✓ · Applebot-Extended ✓
Training ✓ · Search ✓ · User ✓ · Ads ✓
Training ✓ · Search ✓ · User ✓
Search ✓
Search ✓ · User ✓ · Training ✓
Training ✓
Weakest evidence: A-
See each fix below.
Last fetch, hits and errors per bot from your own access logs, verified against each operator's IP list, with spoofed claimants split out. Until then this section shows declared access and synthetic checks only.
▲ Best gain: crawl #13 (+5)
Failing pages per crawl, last 8 crawls. A site-wide check counts once.
What the AI access screen gives for each issue: why it matters, the change to make, and how to check it took. Not part of the health score.
Blocked AI search bots can't fetch your pages, so Perplexity may leave them out of answers. robots.txt line 2: `Disallow: /` (User-agent: PerplexityBot).
User-agent: PerplexityBot Allow: /$ Allow: /app/$ Allow: /about.html$ Allow: /blog/$ Allow: /pricing.html$ Allow: /products/$ Allow: /blog/glazes.html$ Allow: /products/mug.html$ Allow: /products/plate.html$ Allow: /blog/firing.html$
`nosnippet` keeps these pages out of Google Search, AI Overviews & AI Mode and limits what Bing & Microsoft Copilot and Apple (Siri, Spotlight, Safari) can quote from them. Set by the `robots` meta tag. 1 crawled page carries it.
Remove `nosnippet` from the `<meta name="robots">` tag on `/pricing.html`, or remove the tag if that is all it says. If you set it on purpose, mark it as intended in CodoSEO instead.
Sources: https://developers.google.com/search/docs/appearance/ai-features · https://blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat · https://support.apple.com/en-us/119829
Render the main content on the server, or prerender the page, so it is in the HTML. In the HTML as served, this page has fewer than 50 words of text and the app's mount element is empty and a `<noscript>` notice asks for JavaScript. OAI-SearchBot, ChatGPT-User, Claude-SearchBot and 2 more come from the same operators as the crawlers they measured (GPTBot, ClaudeBot and PerplexityBot) and are expected to fetch the same way. A bot that doesn't run JavaScript reads the page as served, not what a browser shows once its scripts have run.
Render the main content of this page on the server, or prerender it at build time, so the text is in the HTML the server sends: `/app/`. Scripts can still run in the browser afterwards: what matters is that the first HTML response already holds the text.