GPTBot Detection: Verify Requests Before Setting Policy

GPTBot detection requires corroborating a claimed user-agent with OpenAI’s current published IP ranges before evaluating the request against a separate training-crawl policy.

Attribute the request before making a training-use decision, without confusing GPTBot with ChatGPT search visibility. This page treats that goal as a documentation and implementation problem: identify the exact product or content behavior, collect route-specific evidence, make the narrowest justified change, and preserve the conditions that limit the conclusion.

The term “GPTBot detection” can invite claims that exceed what a robots rule, content pattern, schema block, or audit record proves. The workflow below keeps access, discovery, interpretation, source quality, eligibility, selection, citation, ranking, and traffic as distinct states.

Readers should be able to use the guidance without accepting a hidden score or vendor label. Each recommendation therefore names the evidence to inspect, the owner to change, the deployment state to validate, and the outcome that remains unmeasured.

Start the GPTBot detection review at the canonical production URL and record the date, response status, effective directives, visible content, and relevant source version. When logs or provider rows are used, retain the attribution method and sampling limits.

A screenshot or search result can motivate investigation but cannot replace page and protocol evidence. Collect the underlying HTML, headers, robots file, structured data, links, or source records required for the specific question.

Filter access logs for the GPTBot token while retaining source IP, request path, status, timestamp, and the full user-agent string. Store the result beside the URL and expected state so another reviewer can reproduce the conclusion without relying on the original operator's memory.

Compare the source IP with OpenAI’s current gptbot.json prefixes because a user-agent string alone can be spoofed. Store the result beside the URL and expected state so another reviewer can reproduce the conclusion without relying on the original operator's memory.

Treat the additional robots.txt marker as evidence of a robots file request, not evidence that GPTBot fetched a content page. Store the result beside the URL and expected state so another reviewer can reproduce the conclusion without relying on the original operator's memory.

Keep GPTBot records separate from OAI-SearchBot and ChatGPT-User because the three agents serve different product actions. Store the result beside the URL and expected state so another reviewer can reproduce the conclusion without relying on the original operator's memory.

• Filter access logs for the GPTBot token while retaining source IP, request path, status, timestamp, and the full user-agent string.
• Compare the source IP with OpenAI’s current gptbot.json prefixes because a user-agent string alone can be spoofed.
• Treat the additional robots.txt marker as evidence of a robots file request, not evidence that GPTBot fetched a content page.
• Keep GPTBot records separate from OAI-SearchBot and ChatGPT-User because the three agents serve different product actions.

Resolve GPTBot detection at the layer that owns the behavior. Content belongs with the page record, crawler policy with the deployed robots configuration, identity with canonical visible profiles, and structured data with the component that renders the corresponding facts.

Write the intended state before editing. That simple contract prevents an optimization request from silently overriding privacy, licensing, access, evidence, or product requirements and gives the validation pass an explicit target.

Allow or disallow GPTBot according to the approved content-use policy. Keep the patch narrow, note the affected routes, and avoid unrelated metadata or navigation changes that make the result harder to attribute.

Place the explicit GPTBot group after reviewing wildcard directives and the most-specific matching rule for representative paths. Keep the patch narrow, note the affected routes, and avoid unrelated metadata or navigation changes that make the result harder to attribute.

Document the training-crawl decision separately from OAI-SearchBot policy for ChatGPT search visibility. Keep the patch narrow, note the affected routes, and avoid unrelated metadata or navigation changes that make the result harder to attribute.

Keep protected content behind access control rather than relying on robots.txt for confidentiality. Keep the patch narrow, note the affected routes, and avoid unrelated metadata or navigation changes that make the result harder to attribute.

• Allow or disallow GPTBot according to the approved content-use policy.
• Place the explicit GPTBot group after reviewing wildcard directives and the most-specific matching rule for representative paths.
• Document the training-crawl decision separately from OAI-SearchBot policy for ChatGPT search visibility.
• Keep protected content behind access control rather than relying on robots.txt for confidentiality.

# Request attribution
User-Agent contains: GPTBot/
Source IP belongs to: https://openai.com/gptbot.json

# Training-crawl policy
User-agent: GPTBot
Disallow: /

Validate GPTBot detection against the deployed canonical route, not only a local component or text fragment. Repeat the original collection method, exercise important variants, and preserve both successful and failed checks.

A technical pass means the intended content, directive, link, identifier, or markup is available under the named conditions. Discovery, selection, citation, ranking, and traffic require separate evidence collected after systems have had time to crawl and process the change.

Fetch the current OpenAI crawler documentation and gptbot.json file, recording the observation date with the log sample. Record the observed value, timestamp, and any measurement gap rather than reducing the result to an unlabeled green check.

Fetch robots.txt from the canonical host and resolve the most-specific matching GPTBot group for representative paths. Record the observed value, timestamp, and any measurement gap rather than reducing the result to an unlabeled green check.

Confirm edge and origin responses match the intended policy without treating a user-agent match as authenticated identity. Record the observed value, timestamp, and any measurement gap rather than reducing the result to an unlabeled green check.

• Fetch the current OpenAI crawler documentation and gptbot.json file, recording the observation date with the log sample.
• Fetch robots.txt from the canonical host and resolve the most-specific matching GPTBot group for representative paths.
• Confirm edge and origin responses match the intended policy without treating a user-agent match as authenticated identity.

Context can make an apparently restrictive, incomplete, or unusual GPTBot detection state intentional. Review the page purpose, audience, policy owner, and evidence freshness before treating it as a defect.

A request that says GPTBot but originates outside the published prefixes remains an unverified claim. Preserve that possibility in the audit record until route-level evidence resolves it.

A robots.txt-marker request is not evidence that a content URL was fetched. Preserve that possibility in the audit record until route-level evidence resolves it.

A GPTBot block does not imply a block on OAI-SearchBot, and robots.txt is not an authentication boundary. Preserve that possibility in the audit record until route-level evidence resolves it.

Claim boundary: A user-agent and published-prefix match supports attribution of the observed request, while a GPTBot directive communicates a training-crawl preference. Neither proves training use, ChatGPT search inclusion, prior-dataset removal, or future model behavior.

This boundary is part of the implementation contract. It must remain visible near the recommendation and in any downstream summary so a machine-readable extract cannot turn technical eligibility into an outcome promise.

OpenAI documents OAI-SearchBot for ChatGPT search, GPTBot for content that may be used to improve generative AI foundation models, and ChatGPT-User for certain user-triggered actions. These names share a vendor, not an outcome. A publisher can allow search discovery while expressing a separate GPTBot training-crawl preference.

ChatGPT-User needs its own boundary. OpenAI says it is not an automatic web crawler, does not determine Search eligibility, and may perform requests for a user even when robots rules would ordinarily disallow an automatic crawler. The policy result and documented product behavior therefore remain separate fields.

The open fixture matrix covers matching specific groups, wildcard fallback, a narrow allow inside a broad disallow, equal-length conflicts, crawler separation, a percent-encoded path, an empty directive, a user-triggered boundary, and an unavailable robots response. A bounded repository evaluator runs all ten inputs and publishes actual results beside the expected contract.

Execution proves only that the named evaluator produced the declared RFC-bound result for each public fixture. Every row still reports request identity as unverified, downstream outcome as unknown, and provider application as not tested or explicitly documented as a boundary. That prevents a parser pass from becoming a claim about live OpenAI behavior.

This guide was reviewed on 2026-08-10 against the primary references linked below. Product roles and documentation can change, so crawler strings, directives, feature status, and policy language should be refreshed before acting on a later release.

The source list supports the factual product or protocol description. Atlas workflow language supplies the evidence boundary; it does not claim private provider access, client outcomes, rankings, citations, or traffic.

1. RFC 9309 Robots Exclusion ProtocolRFC Editor / checked 2026-06-18

   Standards-track robots exclusion protocol reference for crawler access rules and limits.

2. Overview of OpenAI Crawlers — GPTBotOpenAI / checked 2026-08-10

   Current GPTBot role, user-agent form, robots.txt request marker, control boundary, and published IP reference.

3. GPTBot Published IP RangesOpenAI / checked 2026-08-10

   Current OpenAI-published IPv4 prefixes for corroborating claimed GPTBot requests.

4. Robots Exclusion ProtocolRFC Editor / checked 2026-08-10

   Normative user-agent group matching, path precedence, unavailable robots resources, and unreachable robots states.