Should I let OAI-SearchBot and other AI crawlers read my website?
Allow OAI-SearchBot to appear in ChatGPT search results, or block it to limit appearances. The choice depends on which AI systems you want to reach.
By Brian MacDonald ·
Whether to allow AI crawlers depends on your visibility goals. OAI-SearchBot surfaces websites in ChatGPT search results when allowed, but blocking it keeps your site visible as navigational links. Other AI systems have separate crawlers you can manage the same way.
What OAI-SearchBot does
OAI-SearchBot is used to surface websites in ChatGPT's search features. It crawls your site to pull content into ChatGPT's search answers. Other AI systems—Google, Anthropic, Perplexity, others—have their own crawlers that operate the same way.
If you allow it
To help ensure your site appears in search results, allow OAI-SearchBot in your site's robots.txt file and allow requests from OpenAI's published IP ranges. It can take ~24 hours from a site's robots.txt update for the systems to adjust. Other AI platforms follow the same robots.txt standard.
If you block it
Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links. Blocking one crawler doesn't hide your site from search engines: it just removes it from that platform's AI-powered answers.
Appearing in other AI systems
There are no additional requirements to appear in Google's AI Overviews or AI Mode, nor other special optimizations necessary. Google crawls sites the same way its traditional search does. Each AI system has its own crawlers and discovery methods; check their developer documentation for control options.
Publishing content for AI assistants
If you publish content specifically written to answer customer questions, allowing AI crawlers determines whether ChatGPT, Claude, Google and other systems surface it. Ownbox publishes articles on your website written to be found and quoted by AI assistants. Once live, the standard robots.txt rules apply.
Next step
Update robots.txt to allow crawlers from the AI systems you want to reach; allow up to 24 hours for changes to take effect. To have AI assistants find your content, Ownbox publishes articles to your website designed for them to discover and quote.