ChatGPT Search Indexing for Publishers
Discovery, inclusion in summaries, link-only surfacing, training, and user-triggered access are separate policies. A useful implementation preserves those distinctions.
A ChatGPT Search indexing guide for publishers covering OAI-SearchBot, robots.txt, noindex, citations, referral tracking, and verification.
01 / Eligibility
Public and crawlable is the starting state
OpenAI says any public website can appear in ChatGPT Search. To help content be discovered and included in summaries or snippets, the publisher should allow OAI-SearchBot. That is an eligibility and discovery statement, not a submission guarantee. Relevance, source selection, freshness, and the user’s question remain outside the publisher’s direct control.
Check the actual production response. The host must resolve, the URL must return useful content, and robots.txt must not disallow OAI-SearchBot from the path. Important claims should exist in ordinary HTML with stable headings and links. A client-only shell that depends on an interaction or failing API is a weaker publishing surface even if a full browser can eventually display it.
02 / Controls
A crawl disallow and a noindex directive do different work
OpenAI documents that a disallowed page may still appear as a link and title if the URL is obtained from another source and appears relevant. Publishers that do not want that link-only behavior are directed to use noindex. The crawler must be able to fetch the page to read that directive, so blocking the same path in robots.txt can prevent the intended signal from being observed.
This is the same operational trap seen across search systems: request control and index control are not interchangeable. Decide whether the goal is to prevent automated retrieval, prevent surface inclusion, or both. When both matter, confirm the provider’s documented precedence rather than assembling contradictory directives and assuming the stricter-looking one wins.
| Desired state | Crawler access | Page directive |
|---|---|---|
| Eligible for summaries | Allow OAI-SearchBot | Indexable |
| Link-only possibility | Disallow crawler | Directive may be unread |
| Exclude from surfacing | Allow directive fetch | noindex |
| Exclude from training | Separate GPTBot policy | Not an OAI-SearchBot rule |
03 / Citation
Make the source easy to identify and quote accurately
A crawlable page still needs a clear answer. Put the subject and task in the title and H1, define specialized terms, state the direct result before the caveats, and keep those caveats adjacent. Use one canonical URL, visible publication and update dates, author identity, and primary-source links. These are publishing fundamentals, not a secret ChatGPT schema.
Tables are effective when a comparison is the task. Code is effective when a reader can run it. Claims should distinguish observed results from interpretation and identify the date or version they describe. This gives a retrieval system smaller, coherent evidence units while leaving the full page valuable to a human who follows the citation.
04 / Measurement
Track referrals without turning them into citation counts
OpenAI says ChatGPT referral URLs include utm_source=chatgpt.com, which lets analytics isolate visits. Preserve that parameter through redirects and classify it explicitly. Also retain the landing page, date, device, engagement, and conversion definition. A referral is evidence that a user clicked; it is not a full count of answer appearances or mentions.
Server logs can show OAI-SearchBot requests, but a request does not prove indexing or citation. Validate user agents carefully and account for spoofing before reporting bot activity. Manual query checks can capture example citations, yet they are sampled observations affected by location, account state, and result changes. Keep all three evidence streams separate.
05 / Playbook
A safe release sequence for ChatGPT discovery
Define the publisher policy for OAI-SearchBot, GPTBot, and user-triggered agents. Update robots.txt only after the policy is approved. Validate the production file, fetch representative pages with and without JavaScript, confirm canonical and index directives, and make the content source-bound. Add referral classification before launch so the first visits are not lost in an “other” bucket.
After release, watch logs for verified activity, analytics for ChatGPT referrals, and content for staleness. Do not send URLs to Google’s Indexing API, assume IndexNow reaches ChatGPT, or claim an llms.txt file forces inclusion. Those systems have different owners and contracts. Discovery work is successful when the site is eligible, legible, measurable, and honest about uncertainty.
- 01OAI-SearchBot: Search Visibility and Access
OAI-SearchBot robots.txt guidance covering its ChatGPT search role, independent access control, published IP ranges, verification, and claim limits.
- 02GPTBot: Training Control, Not Search Inclusion
GPTBot robots.txt guidance explaining OpenAI model-training controls, its separation from OAI-SearchBot, verification steps, and safe publisher claims.
- 03Write Citable Claims with Visible Evidence Boundaries
Citable claims pair a precise statement with a primary source, observation date, scope, and limitation so readers can inspect what the page actually supports.
- 01OpenAI: Publishers and Developers FAQ
Public-site eligibility, OAI-SearchBot access, link-only behavior, noindex guidance, and ChatGPT referral parameters.
Checked 2026-07-20 - 02OpenAI: Bot and crawler documentation
Current user-agent identities and the separate purposes of OAI-SearchBot, GPTBot, and ChatGPT-User.
Checked 2026-07-20 - 03IETF: Robots Exclusion Protocol
The standards basis, matching behavior, and limitations of robots.txt as a crawler-request protocol.
Checked 2026-07-20