Three different companies, three different crawlers, three different sets of rules, and most advice about "ranking in ChatGPT" skips past all of it. Before any content strategy, there is a mechanical question a site either passes or fails: can the assistant's crawler actually reach the page. Each platform documents its own answer, and they are not the same answer.
This is the part of "LLM SEO" that has nothing to do with writing better content. A page can meet every content-quality standard covered in our GEO vs SEO guide and still never appear in an AI-generated answer, because a robots.txt line written before that platform's crawler existed is quietly blocking it. Checking access is the first thing to get right, not the last.
OpenAI: four bots, not one
OpenAI's own developer documentation names four separate crawlers, each with a distinct job. OAI-SearchBot surfaces sites in ChatGPT's search results and honours robots.txt; blocking it removes a site from that feature specifically. GPTBot crawls content to train OpenAI's foundation models, and also honours robots.txt, blocking it stops training use without necessarily affecting search visibility. ChatGPT-User handles live fetches triggered by an actual user's question or a Custom GPT's action; OpenAI's stated position is that because these are user-initiated, robots.txt rules may not apply to it in the way they do to automated crawling. OAI-AdsBot validates ad landing-page policy compliance and is unrelated to search or training. A site that blocks "GPTBot" in robots.txt to opt out of AI training, a reasonable and common choice, has not necessarily opted out of appearing in ChatGPT's search results, because that runs through OAI-SearchBot instead.
OpenAI's own publisher guidance is direct about the practical requirement: to be discoverable and cited in ChatGPT, a site should not block OAI-SearchBot. If crawling is disallowed, the crawler stops immediately. A disallowed page can, in some circumstances, still surface as a bare link or title if OpenAI has independent signal of its relevance from elsewhere, which a noindex meta tag can prevent, but that is a fallback, not a substitute for being properly crawlable.
Anthropic: three bots, individually controllable
Anthropic documents ClaudeBot (collects web content for model training and development), Claude-User (fetches pages live when a Claude user's query needs web access) and Claude-SearchBot (indexes content specifically for the quality of Claude's own search results). Each can be allowed or disallowed independently in robots.txt, for example disallowing ClaudeBot alone while leaving Claude-User and Claude-SearchBot untouched. Anthropic publishes a source-IP list at claude.com/crawling/bots.json so a site can verify a crawl request is genuinely from Anthropic rather than trust the user-agent string alone, which is worth doing before making an access decision based on server logs.
Perplexity: the same split, different names
Perplexity's documentation describes the same underlying pattern under its own naming. PerplexityBot surfaces and links sites in Perplexity's search results, respects robots.txt, and is explicitly stated not to be used for training Perplexity's models. Perplexity-User fetches pages live when an end user's question triggers it, and Perplexity's own documentation states this fetcher generally ignores robots.txt rules, for the same reasoning OpenAI gives for ChatGPT-User: the fetch was requested by a person, not run as automated crawling. Perplexity also publishes its crawler's source IP ranges for verification.
Checking a crawler is real before acting on it
A user-agent string in a server log is easy to fake, and blocking or allowing access on the strength of one alone is a weak basis for a decision that affects visibility. Anthropic and Perplexity both address this directly: Anthropic publishes its crawlers' source IP ranges at claude.com/crawling/bots.json, and Perplexity publishes PerplexityBot's at perplexity.ai/perplexitybot.json, six specific address blocks in AWS's range as of the last confirmed update to that file. A request claiming to be one of these crawlers from an IP outside the published range is not verified as genuine. OpenAI does not appear to publish an equivalent static IP list in the same way; verification there relies more on the documented user-agent strings and reverse-DNS checks common to other verified crawlers. Checking this before making an access decision based on log analysis is a five-minute task that avoids acting on a spoofed request.
Which index actually powers the answer
This matters more than it first appears, because a platform's search feature is not always built on a crawl index it maintains itself. Reporting on ChatGPT search has previously described Bing as the underlying third-party index for parts of ChatGPT's web search functionality, which would make ordinary Bing indexability an indirect prerequisite for ChatGPT citation in those cases, not a separate, unrelated channel. That specific framing has also shifted: OpenAI is reported to have added a second named third-party search provider to parts of its documentation during 2025, and to be moving toward more in-house indexing over time, so treating any single index as ChatGPT's permanent, sole source of truth is not something to build a long-term strategy on. The practical takeaway holds regardless of which index is in use on a given day: Bing Webmaster Tools registration and indexability are worth having in place alongside Google's, not instead of it, because which platform depends on which index changes without much public notice.
What this means practically
The pattern repeats across all three platforms, which makes the practical checklist shorter than the crawler list suggests:
- Decide separately whether a site wants to opt out of AI training (GPTBot, ClaudeBot) versus opting out of appearing in an assistant's search or answer features (OAI-SearchBot, Claude-SearchBot, PerplexityBot). These are different decisions with different robots.txt entries; blocking one by mistake while intending the other is the most common way a site disappears from an assistant's answers without anyone deciding it should.
- The live, user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) are documented as generally not bound by robots.txt in the same way, because the request came from an actual person's question, not automated crawling. A noindex tag, not a robots.txt disallow, is the documented way to keep a specific page out of that path.
- None of this replaces ordinary crawlability. A page invisible to Googlebot is very likely invisible to these systems too, since several of them rely in part on existing web indices rather than building an entirely separate one from scratch.
What a deliberate robots.txt looks like
Most sites' robots.txt files were last edited before any of these crawlers existed, which means the current setting is usually an accident rather than a decision. A deliberate position looks like naming each crawler explicitly rather than leaving the question unanswered by omission: allowing OAI-SearchBot, Claude-SearchBot and PerplexityBot (the search-facing crawlers) while making a separate, considered choice about GPTBot and ClaudeBot (the training crawlers), rather than one blanket rule covering both. A business that wants to appear in AI-generated answers but does not want its content used to train a foundation model is a coherent, common position, and it requires two different lines in the file, not one. Checking which of these entries currently exist, and whether they say what the business actually intends, is a smaller job than researching the crawlers in the first place, and it is the one most sites have not done.
Measuring it is still the weak point
Google's generative AI performance report, fully rolled out worldwide at the end of August 2026, shows impressions only, no clicks, no queries, broken down by page, country, device and date. Bing's AI Performance report is citations-based: total citations, cited pages, and the grounding queries that led to them, with intent categorisation and citation-share comparisons added as a preview feature in June 2026. Neither OpenAI, Anthropic nor Perplexity publish a first-party citation report for site owners at all as of this research.
Third-party tools have moved into that gap. Profound, Otterly.ai and Peec AI, among others, track brand mentions and citation share across several assistants at once, for a subscription fee, using their own measurement methodology rather than data the platforms publish directly. Treat these the way you would any vendor's proprietary metric, useful for tracking a trend over time, not directly comparable to another tool's number, and not a platform-confirmed figure. The category is young enough that at least one competing tool shut down during 2026, which is worth knowing before committing to one as a long-term reporting system.
The scale, correctly sourced
ChatGPT reached 900 million weekly active users by February 2026, disclosed by OpenAI alongside its funding announcement that month. Google's own investor reporting puts the Gemini app at 950 million monthly active users as of mid-2026. Microsoft's FY26 Q1 earnings call put its Copilot family past 150 million monthly active users across consumer and enterprise use. Separately, Similarweb's own traffic-panel data (not a platform disclosure, their measurement) shows ChatGPT's share of overall generative-AI web traffic falling from roughly 76% in June 2025 to around 53% by May 2026, with Gemini and Claude both gaining share over the same period. Whatever the exact split, the practical conclusion holds: a strategy built around a single assistant is optimising for a shrinking share of a fast-growing category, which is why the crawler-access checklist above is written for all three platforms, not just the largest one.
This is part of what MSD's SEO and GEO service actually checks: crawler access across the platforms that matter to a business's buyers, not just Google, alongside the content and technical fundamentals covered in our SEO and GEO guide. If you do not know which of these bots your site currently allows or blocks, talk to us and bring your robots.txt file.
A few useful questions.
Do I need to block or allow AI crawlers separately from Google?
Does llms.txt help a site show up in ChatGPT or Claude?
Can I measure whether my site is being cited by AI assistants?
Sources and further reading
