AI bots on your site: GPTBot, robots.txt and llms.txt

AI bots are programs that AI companies (OpenAI, Anthropic, Perplexity, Google) use to read websites: to train models, to power search in chatbot answers, or to visit a page at a specific user's request. You can manage them through `robots.txt`, but one rule won't cover them all, because each company runs several bots with different jobs.

In plain words

A bot like GPTBot doesn't buy anything, it only reads pages. The question for a store owner is this: do I want my texts and catalog read to train models, and do I want ChatGPT, Claude or Perplexity to be able to show my store in an answer? These are two different things, and you can decide them separately.

The types of bots

Sources: OpenAI, Anthropic, Perplexity, Google, Google, AI features.

Important: robots.txt is a request, not protection. Google writes that well-behaved crawlers follow it and others may ignore it; truly private information needs a password (Google).

How many there are

Cloudflare estimated the purpose of AI crawling over the 12 months to August 2025: roughly 80% for training, 18% for search, 2% for actions at a user's request. The ratio (January–July 2025 data) of "how many pages a bot crawled per visit it sent back" in July 2025: Anthropic 38,066 to 1, OpenAI 1,091 to 1, Perplexity 195 to 1, Google 5.4 to 1 (Cloudflare, 2025). For the full year 2025, Cloudflare reported that AI bots excluding Googlebot accounted for on average 4.2% of requests to HTML pages, and Googlebot alone for 4.5% (SEJ, December 2025). These figures describe the internet as a whole, not the Ukrainian segment.

Block or allow

There is no single right answer, because this is a question about goals, not technology.

• Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot): if you block them, the store, according to the companies themselves, won't be shown or will be shown worse in those services' answers. Weigh that against how many buyers actually come from chatbots: in an Ahrefs study (June 2025, 81,947 sites) the three main AI sources gave on average 0.19% of a site's traffic, while Google gave 39.35% of all referral traffic (Ahrefs, 2025). There is no such figure for Ukraine.
• Training bots (GPTBot, ClaudeBot, Google-Extended): blocking them harms neither your Google rankings nor your presence in chatbot search, judging by the companies' own descriptions. The decision here is one of principle: whether you are fine with your texts training models.
• Googlebot must not be blocked if you want to be in Google, including its AI answers. Google also asks you not to block it at the CDN and hosting level (Google, AI features).

An example robots.txt that blocks training but keeps the search bots (illustrative, check the bot names in each company's documentation):

``` User-agent: GPTBot Disallow: /

User-agent: ClaudeBot Disallow: /

User-agent: Google-Extended Disallow: /

User-agent: OAI-SearchBot Allow: /

User-agent: PerplexityBot Allow: / ```

Separately: protection services and CDNs may block bots automatically. Check that they haven't blocked the ones you need.

llms.txt: does it work

llms.txt is a file in the site root proposed as a "map" for language models. It is an informal proposal, not a standard.

• Google's guide (May 2026): Google Search does not use llms.txt files, other "files for AI" or special markup; you can create them, but they won't affect visibility in Google Search (Google, 2026).
• SE Ranking (almost 300,000 domains; SEJ report of 20 November 2025): the file was present on about 10% of domains, and no link to how often they are cited in AI was found. The only bot that, according to logs, sometimes requested it was GPTBot, rarely and with no link to citations. This is a correlation analysis, and the authors caution that conclusions depend on the model and the sample (SEJ, 2025).

So there is no evidence of benefit, and Google's guidance says outright that it isn't needed. If you create one, it costs little time; ordinary indexing and site accessibility come first. For more on related myths, see the GEO article.

Common mistakes

• Blocking every bot with "AI" in its name in robots.txt, then wondering why you're missing from chatbot answers.
• Treating robots.txt as protection against copying.
• Blocking Googlebot along with the AI bots.
• Forgetting about CDN and bot-protection settings.
• Spending a week on llms.txt instead of a technical SEO audit.

FAQ

What is GPTBot? An OpenAI bot that collects data that may be used to train models. For showing sites in ChatGPT search there is a separate bot, OAI-SearchBot.

How do I block GPTBot? Add `User-agent: GPTBot` and `Disallow: /` to robots.txt. This means the content should not be used for training.

Do I need to block AI bots? It depends on your goals. Blocking training bots doesn't hurt Google search; blocking search bots may remove you from the answers of the corresponding services.

Does llms.txt work? Google doesn't use it, and an analysis of 300,000 domains found no effect on citations.

Related terms

• GEO (generative engine optimization)
• AI Overviews and AI Mode in Google
• SEO audit of an online store
• Technical site audit

Sources

• OpenAI, Overview of OpenAI crawlers — developers.openai.com/api/docs/bots
• Anthropic, Does Anthropic crawl data from the web — support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
• Perplexity, Perplexity crawlers — docs.perplexity.ai/guides/bots
• Google, Google's common crawlers — developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
• Google Search Central, Introduction to robots.txt — developers.google.com/search/docs/crawling-indexing/robots/intro
• Google Search Central, AI features and your website — developers.google.com/search/docs/appearance/ai-features
• Google Search Central, Optimizing your website for generative AI features (May 2026) — developers.google.com/search/docs/fundamentals/ai-optimization-guide
• Cloudflare, The crawl-to-click gap (29.08.2025) — blog.cloudflare.com/crawlers-click-ai-bots-training
• Search Engine Journal, Cloudflare report: Googlebot tops AI crawler traffic (15.12.2025) — searchenginejournal.com/cloudflare-report-googlebot-tops-ai-crawler-traffic/563303
• Search Engine Journal, llms.txt shows no clear effect on AI citations (20.11.2025) — searchenginejournal.com/llms-txt-shows-no-clear-effect-on-ai-citations-based-on-300k-domains
• Ahrefs, AI traffic study (26.06.2025) — ahrefs.com/blog/ai-traffic-increase