ChatGPT Search Visibility: Crawlers, robots.txt, and Tracking Referral Traffic
Which OpenAI Crawlers Exist - and What Each One Does
OpenAI publishes an overview of its crawlers, and the distinctions between them decide what your robots.txt rules actually control. GPTBot crawls to make generative foundation models more useful and safe - disallowing it means your content stays out of model training. OAI-SearchBot is the search-side crawler, the one that lets pages surface in ChatGPT's search results. ChatGPT-User behaves differently from both: it acts on explicit user actions in ChatGPT and Custom GPTs, does not crawl automatically, and is not used to determine whether content appears in search. There is a fourth for advertisers - OAI-AdsBot validates the safety of pages submitted as ads, visits only those submitted pages, and does not train foundation models. One robots.txt policy cannot serve all four purposes, because visibility in search and inclusion in training are separate decisions.
The Three Access Layers That Silently Block You
OpenAI's guidance for advertisers walks through three layers where access silently dies. First, robots.txt: OpenAI's crawlers respect it, and a disallow stops crawling immediately - the fix is explicit allow rules for the relevant user agents and paths. Second, web protection and bot mitigation: services like Cloudflare or Akamai defend against scraping and can mistake legitimate crawlers for automated traffic, returning 403 Forbidden unless the crawler user agents are allowlisted. Third, human verification: CAPTCHAs, JavaScript challenges, and session validation block automated systems even after robots.txt and the firewall have passed them. The robots.txt allow rules themselves follow the documented pattern:
User-agent: OAI-SearchBot
Allow: /
User-agent: OAI-AdsBot
Allow: /A GPTBot disallow, by contrast, only removes the site from model training - it leaves search visibility untouched. For teams that must allowlist by network, OpenAI publishes stable crawler IP ranges as JSON at openai.com/searchbot.json and openai.com/adsbot.json - and the documentation warns against relying on short-term IP observations from logs, since crawler infrastructure evolves.
Advertiser Landing Pages: A Special Case With a Hard Requirement
Advertisers have a mandatory line in their robots.txt: OAI-AdsBot must be allowed, because OpenAI uses it to validate that pages submitted as ads comply with ad policies. OpenAI also uses landing page content to determine when an ad is most relevant, so the crawler's access shapes delivery, not just compliance. The recommendation is to allow both OAI-AdsBot and OAI-SearchBot - the first keeps the ad campaign reviewable, the second keeps the same domain eligible for organic search visibility. A landing page that passes ad review while blocking OAI-SearchBot has solved half its problem, and the two halves run on different crawlers by design.
Tracking the Traffic Visibility Earns
Visibility only matters once the traffic it sends can be counted. Organic ChatGPT referrals are ordinary browser visits by the time they reach your site, carrying a chatgpt.com referrer - so referrer-based segments in your analytics tool separate them without any extra plumbing. Paid placements - set up in Ads Manager - are measurable in a second way: conversions API delivery ties conversions back to individual ads through the click ID mechanics, which organic citations do not have. Server-side delivery through MOST keeps conversion events flowing to ChatGPT Ads, Meta, TikTok, Reddit, Snapchat, and Pinterest, and the free Pixel Activator validates one landing at pixel.way2.us.
A Checking Order for Visibility Problems
When pages do not appear or ad reviews stall on accessibility, OpenAI's guidance implies a checking order. Read robots.txt for the relevant user agents first - a single disallow stops everything immediately. Then work with infrastructure to check bot mitigation and firewalls for 403 responses against the OpenAI user agents, allowlisting by user agent where possible. Then application-level verification: CAPTCHAs and challenges that exempt humans but not crawlers. Prefer user-agent-based and provider-verified allowlisting over IP-only rules, referencing the published JSON files when IP lists are unavoidable. The conversion tracking validation flow then confirms the traffic, once it arrives, is measured correctly end to end.