Skip to content
Sulayman Bowles / Project Delta

ChatGPT Search Indexing for Publishers

Discovery, inclusion in summaries, link-only surfacing, training, and user-triggered access are separate policies. A useful implementation preserves those distinctions.

A ChatGPT Search indexing guide for publishers covering OAI-SearchBot, robots.txt, noindex, citations, referral tracking, and verification.

01 / Eligibility

Public and crawlable is the starting state

OpenAI says any public website can appear in ChatGPT Search. To help content be discovered and included in summaries or snippets, the publisher should allow OAI-SearchBot. That is an eligibility and discovery statement, not a submission guarantee. Relevance, source selection, freshness, and the user’s question remain outside the publisher’s direct control.

Check the actual production response. The host must resolve, the URL must return useful content, and robots.txt must not disallow OAI-SearchBot from the path. Important claims should exist in ordinary HTML with stable headings and links. A client-only shell that depends on an interaction or failing API is a weaker publishing surface even if a full browser can eventually display it.

02 / Controls

A crawl disallow and a noindex directive do different work

OpenAI documents that a disallowed page may still appear as a link and title if the URL is obtained from another source and appears relevant. Publishers that do not want that link-only behavior are directed to use noindex. The crawler must be able to fetch the page to read that directive, so blocking the same path in robots.txt can prevent the intended signal from being observed.

This is the same operational trap seen across search systems: request control and index control are not interchangeable. Decide whether the goal is to prevent automated retrieval, prevent surface inclusion, or both. When both matter, confirm the provider’s documented precedence rather than assembling contradictory directives and assuming the stricter-looking one wins.

Figure 01 / publisher statesThe desired outcome determines the signals to verify.
Desired stateCrawler accessPage directive
Eligible for summariesAllow OAI-SearchBotIndexable
Link-only possibilityDisallow crawlerDirective may be unread
Exclude from surfacingAllow directive fetchnoindex
Exclude from trainingSeparate GPTBot policyNot an OAI-SearchBot rule

03 / Citation

Make the source easy to identify and quote accurately

A crawlable page still needs a clear answer. Put the subject and task in the title and H1, define specialized terms, state the direct result before the caveats, and keep those caveats adjacent. Use one canonical URL, visible publication and update dates, author identity, and primary-source links. These are publishing fundamentals, not a secret ChatGPT schema.

Tables are effective when a comparison is the task. Code is effective when a reader can run it. Claims should distinguish observed results from interpretation and identify the date or version they describe. This gives a retrieval system smaller, coherent evidence units while leaving the full page valuable to a human who follows the citation.

04 / Measurement

Track referrals without turning them into citation counts

OpenAI says ChatGPT referral URLs include utm_source=chatgpt.com, which lets analytics isolate visits. Preserve that parameter through redirects and classify it explicitly. Also retain the landing page, date, device, engagement, and conversion definition. A referral is evidence that a user clicked; it is not a full count of answer appearances or mentions.

Server logs can show OAI-SearchBot requests, but a request does not prove indexing or citation. Validate user agents carefully and account for spoofing before reporting bot activity. Manual query checks can capture example citations, yet they are sampled observations affected by location, account state, and result changes. Keep all three evidence streams separate.

05 / Playbook

A safe release sequence for ChatGPT discovery

Define the publisher policy for OAI-SearchBot, GPTBot, and user-triggered agents. Update robots.txt only after the policy is approved. Validate the production file, fetch representative pages with and without JavaScript, confirm canonical and index directives, and make the content source-bound. Add referral classification before launch so the first visits are not lost in an “other” bucket.

After release, watch logs for verified activity, analytics for ChatGPT referrals, and content for staleness. Do not send URLs to Google’s Indexing API, assume IndexNow reaches ChatGPT, or claim an llms.txt file forces inclusion. Those systems have different owners and contracts. Discovery work is successful when the site is eligible, legible, measurable, and honest about uncertainty.

Continue through the evidence systemThe article links to narrower Project Delta references for implementation detail and claim boundaries.
  1. 01
    OAI-SearchBot: Search Visibility and Access

    OAI-SearchBot robots.txt guidance covering its ChatGPT search role, independent access control, published IP ranges, verification, and claim limits.

  2. 02
    GPTBot: Training Control, Not Search Inclusion

    GPTBot robots.txt guidance explaining OpenAI model-training controls, its separation from OAI-SearchBot, verification steps, and safe publisher claims.

  3. 03
    Write Citable Claims with Visible Evidence Boundaries

    Citable claims pair a precise statement with a primary source, observation date, scope, and limitation so readers can inspect what the page actually supports.

Primary referencesOfficial and standards sources that bound the article’s claims, with the public review date preserved.
  1. 01
    OpenAI: Publishers and Developers FAQ

    Public-site eligibility, OAI-SearchBot access, link-only behavior, noindex guidance, and ChatGPT referral parameters.

    Checked 2026-07-20
  2. 02
    OpenAI: Bot and crawler documentation

    Current user-agent identities and the separate purposes of OAI-SearchBot, GPTBot, and ChatGPT-User.

    Checked 2026-07-20
  3. 03
    IETF: Robots Exclusion Protocol

    The standards basis, matching behavior, and limitations of robots.txt as a crawler-request protocol.

    Checked 2026-07-20