robots.txt and AI Crawlers: How Not to Block ChatGPT
For AI models to cite your site as a source, robots.txt must not block GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, or PerplexityBot. Open yoursite.com/robots.txt, find the blocks with these names, and remove any Disallow: / line from them. The check takes three minutes, and the fix is usually a single line.
Why AI models need access to your site, and what robots.txt has to do with it
robots.txt is a text file at the root of your site. In it, you tell robots which sections they can crawl and which ones to leave alone. Major companies state that their crawlers read this file and follow its rules.
AI models need your site in two different scenarios. The first is collecting data to train models. The second is looking for sources at the moment a user has already asked a question and the system goes to check live pages.
For businesses, the second scenario matters more. ChatGPT's search relies heavily on the Bing index, Gemini and AI Overviews rely on Google, and Perplexity shows source links right in the answer. If the relevant bots are blocked, your pages never make it into the set the answer is assembled from.
That said, robots.txt works as a request. It's honored by companies that have publicly committed to doing so; actual technical protection of content is handled at the server level, not by this file.
Which AI crawlers visit your site: a list of names
It's worth knowing these bot names by heart — they're exactly what you'll look for in robots.txt and in server logs.
| Crawler | Whose | Why it visits | If blocked |
|---|---|---|---|
| GPTBot | OpenAI | Collects data to train models | Content won't make it into training sets; has a weaker effect on search-based answers |
| OAI-SearchBot | OpenAI | Indexes pages for search inside ChatGPT | Your site won't appear among ChatGPT's search sources |
| ChatGPT-User | OpenAI | Opens a page at a user's or agent's request | ChatGPT won't show the page's content in its answer |
| ClaudeBot | Anthropic | Crawls sites for Claude | Claude won't see your materials when crawling |
| PerplexityBot | Perplexity | Indexes pages for answers with links | Your site won't end up in Perplexity's source block |
| Google-Extended | Controls how content is used in Gemini and AI answers | Doesn't affect regular Google search, but affects inclusion in AI answers |
Google-Extended stands apart: it isn't a separate crawler, but a switch for content Google has already obtained through Googlebot.
How to check your robots.txt in three minutes
- Open yoursite.com/robots.txt in your browser. If it returns a 404, there's no file: there are no restrictions either, but it's still better to create one.
- Press Ctrl+F and search in turn for: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, and then the line User-agent: *.
- Under each block you find, check the Disallow lines. A value of "/" blocks the entire site. An empty value after the colon means crawling is allowed.
- Make sure the file actually sits at the root of the domain. Robots won't apply rules found at yoursite.com/blog/robots.txt.
- Confirm the server returns the file with a 200 status code as plain text, without a redirect to a login page or a placeholder.
Also check your hosting panel and CDN. Many providers have added a "block AI bots" toggle: robots.txt looks open, but the crawler gets a 403 and leaves. If you want to see the picture from the AI models' side right away, run the AI visibility check for your business.
Want to know where your website stands today? Free: an AI-readiness check, a 30-day plan and your first article in 3–6 minutes.
Get my planWhat a robots.txt that opens your site to AI models looks like
For most sites, a single shared block is enough. A minimal working version consists of these lines:
- User-agent: *
- Disallow: — empty value, crawling allowed
- Disallow: /wp-admin/ — service sections can be blocked
- Allow: /wp-admin/admin-ajax.php
- Sitemap: https://yoursite.com/sitemap.xml
Separate blocks for GPTBot and the other bots aren't needed here: the rule for "*" applies to everyone not addressed by a dedicated block. The Sitemap line helps the robot find the full list of pages instead of crawling link by link.
If you want to make the decision explicit for each bot, add a separate block for each: User-agent: GPTBot, followed by an empty Disallow, and the same for OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, and Google-Extended. That way the file clearly shows none of them was overlooked.
Remember the key mechanic: a bot only follows the block addressed to it by name. If there's a dedicated block for GPTBot, it no longer reads the rules under "*". That's exactly how you end up with the common situation where everything is open in the shared block, but a Disallow: / is still sitting in the dedicated one.
Common mistakes that block your site from AI models
Go through the list and check off which of these apply to you.
- Disallow: / left over from development. The site was built on a staging domain, blocked entirely, and nobody reopened it after launch.
- A dedicated block overrides the shared one. There are no restrictions under "*", but the block for GPTBot or PerplexityBot still has a "/".
- A typo in the bot's name. A rule for GPT-Bot or OAI_SearchBot applies to no one: the name has to match exactly, though letter case doesn't matter.
- A robots.txt copied from someone else's site. Along with it come restrictions on sections you don't have, plus a list of blocked AI bots pulled from an article about protecting content.
- Blocked scripts and styles. Blocking CSS and JS folders partially breaks how the page renders for robots that render it.
- The blog got caught in the ban too. Rules like Disallow: /*? or Disallow: /page/ sometimes block pagination and filter pages along with useful content.
- Confusing Disallow with noindex. Disallow blocks crawling a page; noindex blocks it from showing up in results. If you need a page out of the results, crawling is actually left open for it.
- Blocking at the CDN or security-plugin level. The file itself is clean, but bot protection turns the crawler away anyway.
What if I don't want to hand my content over for model training?
That's a reasonable wish, and it can be separated from showing up in answers. The logic is simple: block the training crawlers, leave open the ones that look for sources at answer time.
- Block: GPTBot, ClaudeBot, Google-Extended.
- Leave open: OAI-SearchBot, ChatGPT-User, PerplexityBot.
To be upfront about the limits of this approach: robots.txt doesn't protect content. It gives instructions to those who've agreed to follow them, and it does nothing about parsers that ignore the rules. If content genuinely needs to be locked down, that's a matter of authorization and server-level restrictions.
For a business that wants AI models to recommend it, we recommend opening everything up. You're selling services, not article text: a mention in an answer brings you a customer, while a blocked site simply falls out of consideration. For more on how getting into answers works, see the guide how to get into ChatGPT's answers.
robots.txt is open. What's next?
Open access only lets the robots in. What happens next depends on what they find there: clear titles and descriptions, Schema.org markup in JSON-LD format, FAQ blocks, a working sitemap.xml, and pages that answer customers' questions in customers' own words.
Alongside robots.txt, people often bring up the llms.txt file. It's a format proposed in 2024 (llmstxt.org), not an official standard: support from major AI models isn't guaranteed. It takes about twenty minutes to put together, but don't expect it to substitute for actual content.
We've grouped these checks into 8 technical factors: the site responds, title, meta description, Schema.org JSON-LD, FAQ markup, llms.txt, AI crawlers allowed in robots.txt, sitemap.xml. On insync.com.ua we brought these to 100/100 in our own audit and went from 0 to 24 site citations in ChatGPT in about a week and a half after launching the GEO setup. For category-level queries like "best agency," we're not mentioned yet, which shows that technical readiness opens the door, while content and external mentions do the rest.
All of this is checked and done automatically in GEO Autopilot. You enter your site's address and in 3–6 minutes get a check across 8 factors, a list of who AI models recommend instead of you for your customers' questions, a 30-day plan of 8 articles, and the first article written for your site. This is a free AI visibility audit before any payment, right on the website.
FAQ
Do I need robots.txt if I don't want to block anything?
Yes, it's better to have the file, and it should return a 200 status code. An empty Disallow and a Sitemap line give the robot a clear signal and the sitemap's address. The absence of the file doesn't formally forbid crawling, but having a robots.txt that allows AI crawlers is one of the 8 technical-readiness factors we use to check sites.
What's the difference between GPTBot, OAI-SearchBot, and ChatGPT-User?
According to OpenAI's own description, GPTBot collects data to train models, OAI-SearchBot indexes pages for search inside ChatGPT, and ChatGPT-User opens a specific page at the moment of a user's or agent's request. If you want mentions in answers, the second and third matter most. Training can be blocked separately with its own block.
Will blocking Google-Extended remove my site from regular Google search?
No. According to Google itself, Google-Extended controls content use in Gemini and in AI answers, while regular indexing runs through Googlebot. Organic search isn't affected by this restriction, but inclusion in AI answers may be limited.
I removed the Disallow. When should I expect mentions in ChatGPT?
Nothing happens right away: access is open, but the content still needs to be created and indexed. Indexing new pages in Google usually takes 2–3 weeks, and the first mentions in AI models' answers usually show up after 1–3 months. No one can guarantee "#1 in ChatGPT," which is why we measure results with monthly tracking rather than promises.
How can I tell if AI bots are actually visiting my site?
Check your server logs or CDN analytics and filter visits by user-agent: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot. If there isn't a single such visit in a week, first re-check robots.txt, then check the protection settings on your hosting and CDN. A provider panel's AI-bot-blocking toggle isn't reflected in the file itself at all.