Three agents, three jobs
The names look interchangeable and the consequences are not. Here is each agent in plain language, per OpenAI’s crawler documentation.
OAI-SearchBot is the search crawler. It discovers and fetches pages so they can be surfaced, linked, and summarized in ChatGPT’s search features. Blocking it removes the site from ChatGPT search answers. Allowing it makes the site eligible; it does not make inclusion or citation likely, only possible.
GPTBot is the training crawler. It collects content that may be used to train OpenAI’s foundation models. Blocking it signals that your content should not be used for training. It has no documented role in whether you appear in ChatGPT search.
ChatGPT-User is not a crawler in the scheduled sense. It fetches a page when a user’s action inside ChatGPT or a Custom GPT requires it, for example asking about a specific URL. Because the fetch is user-initiated, OpenAI notes robots.txt rules may not apply. Treat it like a browser you cannot opt out of through robots.txt alone; access controls and authentication remain your levers.
OAI-AdsBot exists too: it validates pages submitted as ChatGPT advertisements. Most publishers can ignore it until they buy ads.
| Agent | What it does | What disallowing it changes | What disallowing it does not change |
|---|---|---|---|
| OAI-SearchBot | Crawls for ChatGPT search surfacing | Site stops appearing in ChatGPT search answers | Training use, user-initiated fetches |
| GPTBot | Crawls for potential model training | Signals content is off-limits for future training | ChatGPT search eligibility, past training data |
| ChatGPT-User | Fetches URLs on a user’s request | Little; robots.txt may not apply | Users can still reference your public pages |
| OAI-AdsBot | Validates submitted ad landing pages | Ad validation for your submitted pages | Search and training policy |
What a robots.txt line actually buys you
A robots.txt rule is a forward-looking instruction to a specific user agent. Three consequences follow that trip people up.
First, the change is not instant. OpenAI’s publisher FAQ says its search systems can take about 24 hours to reflect a robots.txt update. Audit screenshots taken an hour after the deploy prove nothing either way.
Second, blocking is not erasure. The documentation describes what future crawling will respect. It does not describe retroactive removal of content already collected, so do not promise a client that a GPTBot disallow deletes anything from an existing model.
Third, a blocked page can still surface as a bare link. Per the same FAQ, if OpenAI learns a disallowed URL from a third-party search provider or from crawling other pages, it may show just the link and title. The documented way to prevent that is a noindex meta tag, which requires the crawler to be able to read the page. A page that blocks the crawler and relies on a meta tag it cannot fetch has an unenforceable policy.
The decision table
Pick the row that matches the client’s actual intent, then implement exactly that.
| Publisher goal | OAI-SearchBot | GPTBot | Notes |
|---|---|---|---|
| Maximize AI-search presence, no training objection | Allow | Allow | Default for most commercial marketing sites |
| Appear in ChatGPT search, opt out of training | Allow | Disallow | The most requested split; the two rules are independent |
| Exclude the site from both declared uses | Disallow | Disallow | Add noindex on sensitive pages to suppress bare-link surfacing |
| Public marketing open, gated or sensitive paths closed | Path rules | Path rules | Different directories can carry different policies |
Copy-paste robots.txt for each policy
Merge these into the existing file rather than replacing it, and test the final result. A conflicting or more specific group elsewhere in the file can change the effective outcome.
Policy 1: allow both declared uses.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
Policy 2: search yes, training no. This is the configuration most marketing sites ask for.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
Policy 3: exclude both.
User-agent: OAI-SearchBot
Disallow: /
User-agent: GPTBot
Disallow: /
Policy 4: open marketing site, closed sensitive paths.
User-agent: OAI-SearchBot
Disallow: /portal/
Disallow: /drafts/
User-agent: GPTBot
Disallow: /portal/
Disallow: /drafts/
Remember what none of these blocks: ChatGPT-User fetches triggered by a person, and any crawler that ignores robots.txt. Robots.txt is a published policy, not an access control. Anything genuinely confidential belongs behind authentication.
Verify the policy like an auditor
Configuration is a claim; verification is evidence. After any change:
- Fetch the public robots.txt without authentication and archive the body, status, and timestamp.
- Evaluate each agent’s group against every priority URL, not just the homepage.
- Confirm the CDN, WAF, or bot-management layer is not blocking an agent you allowed. Security products frequently block AI crawlers by default, which silently defeats a search-visibility objective.
- If you have log access, verify claimed crawler hits against OpenAI’s published IP ranges (searchbot.json, gptbot.json, chatgpt-user.json) before treating them as real. User-agent strings are trivially spoofed.
- Recheck about 24 hours later, and again after any CDN or security-rule deployment.
Keep the evidence labeled honestly: a robots.txt capture proves the served policy, a verified log hit proves a fetch, and neither proves inclusion, citation, or training use. That separation between access and outcomes is the first layer of the four-layer visibility model, and the full access review sits inside the 47-point audit checklist.
Seven of the 47 checks cover technical eligibilityPublic responses, robots rules, indexability, canonicals, rendered text, structured facts, and a date on every one of them. Confirm what you verified and the checker ranks the rest of the protocol against it.
Score the access reviewThis page covers OpenAI’s agents specifically. Google takes the opposite architecture, with AI Overviews and AI Mode riding the normal search index and the standard snippet controls, sorted requirement by requirement in the AI Overview technical requirements guide and covered structurally in GEO vs SEO. For the reporting side of Microsoft’s ecosystem, see Bing Webmaster Tools AI performance.