GetMentionedByAIOAI-SearchBot vs GPTBot: which one should robots.txt allow?
OAI-SearchBot powers ChatGPT search visibility; GPTBot is used for model training. Here is how to control each one without blocking the wrong bot.
OAI-SearchBot and GPTBot are separate OpenAI crawlers with separate controls. OAI-SearchBot is used to surface websites in ChatGPT search results. GPTBot collects web content that may be used to improve OpenAI’s foundation models. You can allow the first and block the second.
Short answer: if ChatGPT search citations matter to you, allow OAI-SearchBot. Decide separately whether you want to allow GPTBot for training.
What is the difference between OAI-SearchBot and GPTBot?
| User agent | Job | What blocking changes |
|---|---|---|
| OAI-SearchBot | Indexes pages for ChatGPT search | Your pages are not shown in ChatGPT search answers, though navigational links may still appear |
| GPTBot | Collects content for model training | Signals that future crawls should not use your content for training |
| ChatGPT-User | Fetches a page after a user request | Not an automatic crawler; access behavior is separate from the two bots above |
OpenAI explicitly says each setting is independent. That matters because many copied robots.txt templates put every AI user agent under one blanket block. A policy intended to opt out of training can therefore remove the same site from live ChatGPT search.
A practical robots.txt policy
Start with the business decision, not the syntax. Publishers that want search referrals but do not want training use can allow OAI-SearchBot and disallow GPTBot. Teams that are comfortable with both uses can allow both. Teams that want neither can block both, accepting the search-visibility trade-off.
- Allow OAI-SearchBot when appearing in ChatGPT search answers is desirable.
- Block GPTBot only when your policy is to opt out of OpenAI model-training crawls.
- Keep the groups separate so a later edit does not change both policies accidentally.
- Check CDN, WAF, and bot-management rules too; an allowed robots.txt group cannot override an edge block.
How to verify what your site currently allows
Run the free AI crawler checker. It fetches the live robots.txt, applies the matching user-agent group, and shows the exact Allow or Disallow rule. A manual file review is still useful, but the matching rule matters more than a line that merely contains the bot name.
After changing robots.txt, do not expect an immediate citation jump. OpenAI says its systems may take about 24 hours to adjust to an OAI-SearchBot change. Crawl permission only creates eligibility; relevance, indexing, source quality, and the user’s query still decide whether a page is used.
Primary source
OpenAI maintains the current user-agent strings, published IP ranges, and control descriptions in its official crawler documentation. Re-check that page before shipping a permanent security policy because crawler names and versions can change.
Common questions
Does blocking GPTBot remove my site from ChatGPT search?
No. GPTBot is the training crawler. OAI-SearchBot is the separate crawler used for ChatGPT search visibility.
What happens if I block OAI-SearchBot?
OpenAI says opted-out sites will not be shown in ChatGPT search answers, although navigational links may still appear.
Can I allow OAI-SearchBot and block GPTBot?
Yes. OpenAI documents the controls as independent.