- OAI-SearchBot is an OpenAI crawler used to help discover and surface web content in ChatGPT search experiences. It is distinct from GPTBot, which crawls for potential use in improving OpenAI’s generative AI foundation models. The two have separate user-agent strings and serve separate functions.
- Blocking OAI-SearchBot in robots.txt prevents OpenAI from crawling your content through OAI-SearchBot and can limit its ability to surface page content in ChatGPT search. However, OpenAI notes that a URL and title may still sometimes appear if the page is discovered through third-party search providers or other sources. Sites can be blocked inadvertently through wildcard disallow rules or broader bot-management configurations intended for other crawlers.
- Allowing OAI-SearchBot requires either no robots.txt rule for it (defaults to allow) or an explicit
Allow: /directive. No submission process exists — the crawler discovers your site through links from crawled pages and other web-discovery pathways. - You can block GPTBot (training data) while allowing OAI-SearchBot (ChatGPT search) — these are independent decisions. Many publishers choose this configuration to opt out of GPTBot crawling while permitting OAI-SearchBot access.
- Verify crawl access by checking your server logs for verified OAI-SearchBot requests (cross-referenced against OpenAI’s published IP ranges) and confirming your robots.txt configuration at yourdomain.com/robots.txt.
Methodology note: OAI-SearchBot specifications are based on OpenAI’s published crawler documentation at platform.openai.com/docs/bots. Behaviour observations are based on server log analysis and community testing as of mid-2026. OpenAI has not published a detailed technical specification of how OAI-SearchBot prioritises crawl targets, determines crawl frequency, or selects content for use in ChatGPT search responses.
What OAI-SearchBot Is
OAI-SearchBot is an OpenAI crawler used to help discover and surface web content in ChatGPT search experiences. Publishers who want their public content to be available through this crawler should allow OAI-SearchBot in robots.txt and permit requests from OpenAI’s published IP ranges. OpenAI does not publicly document the complete retrieval, indexing, or citation-selection architecture behind ChatGPT Search.
OAI-SearchBot and GPTBot are two distinct OpenAI crawlers with different documented purposes. The distinction matters for how you configure access:
| Crawler | Purpose | User-agent | robots.txt directive |
|---|---|---|---|
| OAI-SearchBot | Web crawling used to help discover and surface content in ChatGPT search experiences | OAI-SearchBot |
User-agent: OAI-SearchBot |
| GPTBot | Web crawling that may be used to improve generative AI foundation models | GPTBot |
User-agent: GPTBot |
Allowing OAI-SearchBot permits OpenAI’s search crawler to access eligible public content on your site. This supports crawler-based discovery but does not guarantee that a page will be retrieved, cited, or shown in a ChatGPT response. Allowing GPTBot permits crawling through GPTBot for potential use in improving OpenAI’s generative AI foundation models. These are separate decisions with separate consequences.
OAI-SearchBot Technical Details
Confirmed specifications from OpenAI’s published documentation:
- User-agent name:
OAI-SearchBot - Full user-agent string:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot - IP range: OpenAI publishes its IP ranges at openai.com/searchbot (and in the crawlerIPs.json file it references)
- Crawl purpose: Helping discover and surface web content in ChatGPT search experiences
- robots.txt compliance: OpenAI states that OAI-SearchBot respects robots.txt directives
How to Check Your Current robots.txt
Your robots.txt file is located at the root of your domain: https://yourdomain.com/robots.txt. Navigate there directly in your browser to see its current contents.
Look for any of these patterns that may be blocking OAI-SearchBot:
Pattern 1: Wildcard block (blocks everything including OAI-SearchBot)
User-agent: *
Disallow: /For a crawler, the robots.txt group with the most specific matching user-agent token applies. A dedicated User-agent: OAI-SearchBot group therefore governs OAI-SearchBot instead of the generic User-agent: * group. If your site uses a wildcard block, add an explicit OAI-SearchBot allow in its own user-agent group.
Pattern 2: Explicit OAI-SearchBot block
User-agent: OAI-SearchBot
Disallow: /This deliberately blocks OAI-SearchBot from the entire site. Remove or modify this rule if you want to permit crawler-based discovery for ChatGPT Search.
Pattern 3: GPTBot block (does NOT block OAI-SearchBot)
User-agent: GPTBot
Disallow: /This blocks GPTBot but does not affect OAI-SearchBot. This is a common and intentional configuration — opting out of GPTBot crawling while permitting OAI-SearchBot access.
How to Configure robots.txt for Your Use Case
Option A: Allow both OAI-SearchBot and GPTBot (fully open to OpenAI)
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /Or simply: do not include any rule for either crawler, which defaults to allow for both.
Option B: Allow OAI-SearchBot (ChatGPT Search) but block GPTBot (training data)
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /This is the configuration for publishers who want to permit OAI-SearchBot crawler access while opting out of GPTBot crawling. The two directives are independent.
Option C: Block both OAI-SearchBot and GPTBot
User-agent: OAI-SearchBot
Disallow: /
User-agent: GPTBot
Disallow: /This opts out of both OAI-SearchBot crawling and GPTBot crawling. Blocking OAI-SearchBot stops future access through that crawler and limits how OpenAI can discover and surface your content, but OpenAI notes that a URL and title may still sometimes appear if the page is discovered through third-party search providers or other sources.
Option D: Allow OAI-SearchBot for specific paths only
User-agent: OAI-SearchBot
Disallow: /members/
Disallow: /private/
Allow: /This allows OAI-SearchBot to crawl public content while blocking it from member-only or private sections. Useful for sites with both public editorial content and gated member areas.
For each crawler, the robots.txt group with the most specific matching user-agent token applies. A
User-agent: * group does not override a specific User-agent: OAI-SearchBot group — the specific group governs that crawler. To allow OAI-SearchBot despite a wildcard block, add a dedicated User-agent: OAI-SearchBot group with Allow: /. That group’s rules apply to OAI-SearchBot regardless of its position relative to the wildcard group in the file. If the OAI-SearchBot group also contains narrower Disallow rules, URL-path matching specificity still applies; Allow: / should not be treated as a universal override of every more specific path rule within that group.
How OAI-SearchBot May Discover Your Site
There is no submission process for OAI-SearchBot equivalent to Google Search Console’s sitemap submission. Publishers should make important public pages technically crawlable, maintain strong internal linking, and avoid blocking OAI-SearchBot if they want crawler-based discovery.
ChatGPT Search can also use third-party search providers and other web-discovery pathways. Maintaining conventional search-engine discoverability, including Bing where relevant, is therefore sensible, but OpenAI does not document Bing indexation or Bing Webmaster Tools submission as a direct mechanism for accelerating OAI-SearchBot crawling.
Crawler Access, Discovery, and Citation: What Each Step Proves
Allowing OAI-SearchBot and seeing it in your server logs are two different signals. This table separates what each actually establishes:
| Signal | What it proves | What it does not prove |
|---|---|---|
| robots.txt allows OAI-SearchBot | Crawler is not disallowed by that robots rule | OpenAI has crawled the URL |
| Verified server-log request | OAI-SearchBot requested the URL | The URL will be cited or surfaced |
| HTTP 200 response | Request successfully received content | Content will be retrieved for a query |
| ChatGPT citation observed | Source appeared in that sampled response | OAI-SearchBot was necessarily the discovery path |
| No observed crawler requests | No requests visible in selected logs or window | Site is definitely blocked or undiscovered |
How to Verify OAI-SearchBot Is Crawling Your Site
Method 1: Server access logs
OAI-SearchBot visits are recorded in your server access logs as any other crawler visit. Filter your access log for the OAI-SearchBot user-agent string:
grep "OAI-SearchBot" /path/to/access.logFor Apache or Nginx servers, the log format typically records the full user-agent string. Look for entries containing OAI-SearchBot/1.0. Cross-reference the source IP against OpenAI’s published IP ranges to confirm the request is from OpenAI’s infrastructure rather than a spoofed user-agent. Absence of requests during a selected period does not prove that the site is blocked or undiscovered; crawl cadence is unpublished and your logging or CDN configuration may also affect what you observe.
Method 2: Robots.txt validation
Navigate to https://yourdomain.com/robots.txt in your browser and confirm that no rule blocks OAI-SearchBot. The absence of a blocking rule is sufficient to allow crawl — you do not need an explicit allow directive.
Method 3: Test ChatGPT citations manually
Ask ChatGPT about topics you cover with specific, authoritative content and check whether your domain appears as a cited source. A manual citation test confirms that your domain appeared as a source in that sampled response. It does not reveal whether OAI-SearchBot, a third-party search provider, or another retrieval pathway was responsible. For the broader picture of what each signal establishes, see the table above.
For how to track ChatGPT referral traffic in GA4 and measure which landing pages receive measurable visits, see ChatGPT Search SEO: How to Improve Content Discovery and Citation Visibility. For what OpenAI has and has not documented about citation selection, see ChatGPT Search Ranking Factors and Sources Cited.
Method 4: Cloudflare or CDN bot analytics
Cloudflare recognises OAI-SearchBot as an AI Search bot and provides AI crawler controls. Whether requests are allowed, challenged, or blocked depends on your Cloudflare bot, WAF, and AI crawler-control configuration. If robots.txt permits OAI-SearchBot but no verified requests reach the origin, inspect Cloudflare security events and AI crawler settings before assuming OpenAI has not discovered the site.
Common Blocking Scenarios and Fixes
| Blocking Scenario | How to Identify | Fix |
|---|---|---|
| Wildcard block in robots.txt | User-agent: * / Disallow: / in robots.txt with no specific OAI-SearchBot user-agent group |
Add a dedicated User-agent: OAI-SearchBot group with Allow: /. Group precedence is based on user-agent specificity, not position in the file. |
| Explicit OAI-SearchBot block from a prior decision | User-agent: OAI-SearchBot / Disallow: / in robots.txt |
Remove the disallow rule or change to Allow: / |
| Cloudflare bot or WAF rules blocking AI crawlers | No verified OAI-SearchBot entries in access logs despite allow in robots.txt; Cloudflare Firewall Events showing bot blocks | Configure bot/WAF rules using Cloudflare’s verified-bot classification or validate requests against OpenAI’s published IP ranges. Do not allowlist solely on the OAI-SearchBot user-agent string — user-agent strings can be spoofed. |
| Web Application Firewall (WAF) rules blocking bots | Access log shows no verified OAI-SearchBot visits; WAF logs show blocked requests | Add OAI-SearchBot to the WAF allowlist, validating against OpenAI’s published IP ranges rather than user-agent string alone |
| noindex meta tag | <meta name="robots" content="noindex"> on key pages |
A noindex directive is distinct from robots.txt crawl permission. OpenAI does not publicly document how noindex directives affect ChatGPT Search retrieval or citation behavior. Use OAI-SearchBot-specific robots.txt controls when your intent is to control this crawler. |
| Content behind JavaScript rendering | Page content only visible after JS execution | OpenAI does not publish a complete rendering specification for OAI-SearchBot. For critical public content, server-rendered or static HTML is the safer implementation because it does not depend on client-side JavaScript execution for crawler access. |
Cloudflare recognises OAI-SearchBot as an AI Search bot in its bot reference and provides AI crawler controls. However, whether requests are allowed, challenged, or blocked depends on each site’s Cloudflare bot, WAF, and AI crawler-control configuration — not on a universal default. If you use Cloudflare and OAI-SearchBot is not appearing in access logs despite robots.txt allowing it, check Cloudflare security events and AI crawler settings. CDN-level blocking takes precedence over robots.txt rules because the request never reaches the origin server. Configure exceptions using Cloudflare’s verified-bot classification or OpenAI’s published IP ranges — do not rely on user-agent matching alone.
OAI-SearchBot and Crawl Rate
OpenAI does not publish crawl rate data or allow publishers to set a crawl rate limit for OAI-SearchBot in robots.txt (the Crawl-delay directive is not supported by all crawlers and may not be honoured by OAI-SearchBot). If OAI-SearchBot is generating excessive server load:
- Use IP-based rate limiting at the server or CDN level, targeting OpenAI’s published IP ranges. Use OpenAI’s current published ranges and refresh your rules when those ranges are updated — avoid hard-coding a static list indefinitely.
- Review OpenAI’s current OAI-SearchBot documentation at openai.com/searchbot for any available crawl-support or contact guidance
- As a last resort, blocking OAI-SearchBot in robots.txt stops future access through that crawler and limits how OpenAI can discover and surface your content — use it only when that tradeoff is intentional
Controlling or Requesting Removal of Content From OpenAI Search Experiences
If you want specific pages or your entire site removed from ChatGPT Search experiences:
- Block via robots.txt: Add a
Disallowrule for OAI-SearchBot — this prevents future crawl through this crawler but does not immediately affect content already surfaced through other pathways - OpenAI removal request: Where OpenAI provides a content-removal request process, submit the relevant URLs or domain through the current official form, linked from openai.com/searchbot. A removal request is separate from robots.txt crawler controls and does not constitute an immediate or universal deletion guarantee.
- noindex meta tag: A
<meta name="robots" content="noindex">tag signals to search indexing bots that a page should not be indexed. OpenAI does not publicly document hownoindexdirectives affect ChatGPT Search retrieval or citation behavior. Use OAI-SearchBot-specific robots.txt controls when your intent is to stop crawler access.
Note that stopping future crawl, removal from ChatGPT search experiences, and model training controls are different mechanisms. Use the robots.txt Disallow directive to control OAI-SearchBot crawler access; use the OpenAI removal form for requesting removal from search experiences; use GPTBot controls separately if your intent is to limit GPTBot crawling.
Frequently Asked Questions
Do I need to do anything special to allow OAI-SearchBot, or is it allowed by default?
By default, crawlers are allowed if there is no robots.txt rule blocking them. If your robots.txt has no rule for OAI-SearchBot and no wildcard block that would apply, OAI-SearchBot is already allowed. Check your robots.txt to confirm there is no blocking rule — that is all you need to do. No submission or registration process is required.
I blocked GPTBot to opt out of training. Does that automatically block OAI-SearchBot too?
No. GPTBot and OAI-SearchBot are separate user-agents with separate robots.txt directives. A User-agent: GPTBot / Disallow: / rule applies to GPTBot only. OAI-SearchBot is unaffected by a GPTBot rule. Confirm by checking your robots.txt for any rule containing OAI-SearchBot.
How long after allowing OAI-SearchBot will my site appear in ChatGPT Search?
OpenAI publishes no guaranteed timeline between allowing OAI-SearchBot and observing crawler activity or citations. Crawl timing and citation behavior are not publicly specified. Verify technical access, maintain normal web discoverability, and monitor server logs for observed OAI-SearchBot requests rather than assuming a fixed discovery timeline.
Can OAI-SearchBot access pages protected by HTTP authentication?
OAI-SearchBot cannot ordinarily retrieve content that requires authentication credentials it does not possess. Therefore, authenticated content should not be assumed to be directly crawlable through OAI-SearchBot. Other licensed or provider pathways, where applicable, are separate.
Is there a way to verify that OAI-SearchBot’s IP is legitimate before allowing it?
Yes. OpenAI publishes its IP ranges for OAI-SearchBot at https://openai.com/searchbot. Cross-reference the request IP against OpenAI’s currently published OAI-SearchBot IP ranges to confirm visits are from OpenAI’s actual infrastructure. Do not trust the user-agent string alone — user-agent strings can be spoofed.
Does OAI-SearchBot respect the Crawl-delay directive in robots.txt?
OpenAI has not confirmed support for the Crawl-delay directive. Because support varies by crawler, do not rely on this directive as your primary OAI-SearchBot rate-control mechanism; use validated server or CDN controls where necessary.
How This Fits Into ChatGPT Search Optimisation
OAI-SearchBot access is a technical prerequisite for discovery through OAI-SearchBot specifically; it is not the same as overall ChatGPT citation eligibility or content quality. For the broader optimisation picture:
- ChatGPT Search Ranking Factors — what OpenAI has documented and what remains unconfirmed about citation selection
- ChatGPT Search SEO — content, structure, and measurement practices for improving citation visibility
- ChatGPT Search vs Google Search — how the two ecosystems differ in crawler controls, indexing model, and measurement
Sources
TL;DR OAI-SearchBot is an OpenAI crawler used to help discover and surface web content in ChatGPT search experiences. It is distinct from GPTBot, which crawls…
