Allow the AI search and answer crawlers in robots.txt
These crawlers decide whether a page can appear in ChatGPT search, Claude search and Perplexity answers. OpenAI recommends allowing OAI-SearchBot for search inclusion, Anthropic says blocking Claude-SearchBot prevents indexing for search, and Perplexity says PerplexityBot is the crawler that surfaces and links sites. While they are blocked, nothing else in this plan can work.
- CONFIRMED OpenAI: Overview of OpenAI crawlers
- CONFIRMED Anthropic: Does Anthropic crawl data from the web?
- CONFIRMED Perplexity: Perplexity crawlers
How to do it
- Open robots.txt and list every User-agent group with its Disallow lines.
- Remove Disallow rules that cover your key pages for OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot and Googlebot, and for the user-triggered fetchers ChatGPT-User and Claude-User. If a User-agent: * group blocks everything, add an explicit group for each of these agents with Allow: /.
- Perplexity-User and ChatGPT-User fetch pages when a person asks, and their vendors say robots.txt may not apply to them; for those two, your CDN setting (the next action) is what decides access.
- Keep private areas (admin, cart, staging, internal search) disallowed for everyone.
- Add a Sitemap: line with the full sitemap URL on your canonical host, for example https://yoursite.com/sitemap.xml.
- Publish the file, then allow about a day: OpenAI says robots.txt changes take about 24 hours to reach its search.
How to check it worked
- Open the live robots.txt in a private window: each agent above has either no group of its own (so User-agent: * applies) or a group with no Disallow covering your key pages.
- Google Search Console's robots.txt report shows the version Google fetched, with no errors.
Answers question 01: Does your robots.txt let the AI search and answer crawlers in?