Which AI Bots Crawl My Website and Should I Block Them?

AI crawlers are programs that AI companies send to fetch web pages. Most do one of three jobs. They collect pages to train a model, build an AI search index, or load a page a person asked an assistant about.

If you want your business to show up in AI answers, allow the AI search crawlers such as OAI-SearchBot and PerplexityBot, and decide on training bots such as GPTBot and ClaudeBot separately. Each kind costs you something different to block.

AI crawlers fetch pages for AI companies

AI crawlers are automated programs, such as GPTBot and ClaudeBot, that request web pages for an AI company and identify themselves with a name called a user agent.

Your site’s robots.txt file uses those names to set rules for each bot. The bot decides whether to follow them, because nothing in the file can force it.

Most AI crawling is for training, and it sends few visitors back. Cloudflare reported in August 2025 that 80 percent of AI crawling on its network over the previous 12 months was for training. In July 2025, Anthropic’s crawlers took about 38,000 pages for each visit Claude referred to a site, and OpenAI’s took 1,091 for each visit it referred (Cloudflare, August 29, 2025).

A few crawlers do more than one job. Applebot feeds Apple’s search features and may also feed training. Amazonbot improves Amazon’s products and may train its models. meta-externalagent collects for Meta’s training and product indexing. Blocking one of these blocks every job it does.

Two names in the table below, Google-Extended and Applebot-Extended, are control tokens, which let a site say whether its content may be used for AI training. No crawler sends them, so you’ll never see them in your logs.

Being cited in an AI answer and being used in training are separate things, and What Is AI? explains why. Browser agents are a fourth kind of AI traffic, covered in Is Your Website Ready for AI Agents?, and What Is an AI Agent? explains how they differ from crawlers.

Here are the AI crawlers each company runs

OpenAI, Anthropic, Google, Perplexity, Meta, Apple, Amazon and Common Crawl all document their AI bots, and most run a separate bot for each job. This table is current as of October 30, 2026. Your logs will show other names too.

Every “Follows it” is the operator’s own claim. Whether a request came from that bot is a separate check.

Microsoft controls Bing’s AI use with page tags. In a September 2023 post, Bing said a noarchive tag keeps a page out of Bing Chat, since renamed Copilot, and out of training, and nocache limits answers to the URL, title and snippet. Both keep the page in regular results. Check Bing’s current guidelines before relying on that.

What are GPTBot, OAI-SearchBot and ChatGPT-User?

OpenAI runs GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for pages a person asks ChatGPT to open.

Blocking GPTBot tells OpenAI not to use your content to train its models. A site that blocks OAI-SearchBot won’t be shown in ChatGPT search answers, though OpenAI’s crawler page says it “can still appear as navigational links.”

The two bots are independent, so you can block one and allow the other. If you allow both, OpenAI says it may use one crawl for both purposes. ChatGPT-User only runs when a person asks, and OpenAI says robots.txt “rules may not apply” to it.

What are ClaudeBot, Claude-SearchBot and Claude-User?

Anthropic runs ClaudeBot for training, Claude-SearchBot for search, and Claude-User for pages a person asks Claude to read, and it says all three follow robots.txt.

Anthropic’s crawler page says blocking ClaudeBot signals that your future material should be left out of training. Blocking Claude-SearchBot stops indexing for search and “may reduce” your visibility in Claude’s results. Blocking Claude-User stops Claude from retrieving your pages for user questions, which Anthropic says may also reduce your visibility.

Block lists copied around the web often carry anthropic-ai and Claude-Web, which Anthropic’s page doesn’t list. A file that names only those two leaves all three current bots to the * group, the default rules for any bot the file doesn’t name.

What is Google-Extended, and which Google crawlers matter for AI?

Google-Extended is a robots.txt token that governs Gemini training and Gemini app answers, Googlebot feeds Search including AI Overviews and AI Mode, and a Search Console setting controls AI Overviews and AI Mode.

Google-Extended has a cost that’s easy to miss. It also governs grounding, the step where Gemini pulls in pages to support an answer, and Google’s October 30 post says sites that “specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.” The post doesn’t say whether a file that never names Google-Extended counts as allowing it. Google says the token “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.”

The Search generative AI setting removes a site’s links and content from AI Overviews, AI Mode and the AI features in Discover. Google says “This control doesn’t affect AI training” (Search Console Help). Each Search Console property has its own setting, a child property can override its parent, and Google says content is excluded within one to two days, sometimes longer. The site has to be verified in Search Console first.

Google-Agent browses and takes actions a user asks for, and Google says its user-triggered fetchers “generally ignore robots.txt rules.”

What are PerplexityBot and Perplexity-User?

Perplexity runs PerplexityBot to index pages for its search results and Perplexity-User to load a page when a person asks a question.

PerplexityBot isn’t used to train AI foundation models, according to Perplexity’s crawler page, and the company recommends allowing it. Perplexity-User “generally ignores robots.txt rules,” because a person requested the fetch.

Which bots do Meta, Apple and Amazon run?

Meta’s main AI bots are meta-externalagent, meta-webindexer and meta-externalfetcher, Apple runs Applebot with the Applebot-Extended training token, and Amazon runs Amazonbot, Amzn-SearchBot and Amzn-User.

On Meta’s crawler page, meta-externalagent crawls for uses that include “training foundation AI models.” meta-webindexer is the one tied to citations, since Meta says allowing it “helps us cite and link to your content in Meta AI’s responses.” meta-externalfetcher handles “individual links at a user’s request” and “may bypass robots.txt rules.”

Meta’s page doesn’t say whether blocking meta-externalagent affects those citations.

If your business accepts Meta’s business terms, for example to run ads, its Commercial Terms preview authorizes Meta to crawl the domains “associated with your products or services” for advertising and business products from November 20, 2026. The clause doesn’t mention robots.txt, and it’s unclear how the terms and the file interact.

Applebot crawls for Siri, Spotlight and Safari, and Apple’s support page says its data “may also be used to help train Apple foundation models.” Blocking Applebot-Extended opts your content out of that training, and Apple says those rules “are not considered in ranking for Search.”

Amazon’s crawler page says Amazonbot improves its products and may train its models, while Amzn-SearchBot “does not crawl content for generative AI model training.” Amazon’s training opt-out is a tag in each page’s head, <meta name="robots" content="noarchive">, which most sites can add through their SEO settings.

What is CCBot?

CCBot is the crawler for Common Crawl, a nonprofit that publishes a free archive of the web that AI developers have used to train models. OpenAI’s GPT-3 paper lists filtered Common Crawl as 60 percent of its training mix (Brown et al., 2020, table 2.2).

What Is AI? covers where AI training data comes from.

Blocking CCBot stops future crawling, and Common Crawl’s FAQ says it follows the file. Its CCBot page doesn’t say a block removes copies already in the archive. Whether those copies should stay is contested, and in June 2026 the publisher group Digital Content Next asked Common Crawl to remove member content it had already collected.

What you lose depends on which AI bot you block

Blocking a training-only AI bot asks its company not to train on your future pages, and blocking a search bot can keep you out of that company’s AI answers. Some user-triggered fetchers follow a block and others may skip it, depending on the company.

Most sites should allow AI search crawlers

A site that wants to be cited in AI answers should allow the AI search crawlers and treat training as a separate decision, because blocking a search crawler can keep the site out of answers its customers read.

If you run a plumbing business, an online store or a software company and want to be found, allow Googlebot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, Applebot and Amzn-SearchBot. Blocking GPTBot, ClaudeBot and CCBot doesn’t touch those search crawlers.

The robots.txt example below opts out of the main training bots and stays in AI search. The readiness guide weighs each kind of AI traffic, including browser agents.

Once the bots can reach you, the Answer Engine Optimization (AEO) guide covers how to get retrieved and cited, including where ChatGPT, Perplexity and Google get their answers.

Should you block AI training bots?

If your writing is the product, as it is for a publisher, an online course or a paid newsletter, you can block the training bots and tokens and keep the search crawlers if citations send you readers. Paid content belongs behind a login, because robots.txt can’t lock a page.

Three questions help with the training decision.

• Is your writing the thing you sell?
• Would you mind a model learning from it with no link back to you?
• Is the page public anyway, and already quoted elsewhere?

A yes to the first two points toward blocking the training bots. A yes to the third means a block protects less, because copies on other sites can reach a model.

Can AI bots raise your hosting bill?

AI bots can raise your hosting bill and slow your site. Read the Docs reported that one crawler cost it over $5,000 in bandwidth in May 2024. IP-based rate limits failed because the crawlers used many addresses, and blocking them cut its file bandwidth by 75 percent (Read the Docs, July 25, 2024). If that happens to you, start with your CDN’s bot rules.

AI crawlers follow their own robots.txt group

AI crawlers read robots.txt, a plain text file at the root of a site, by finding the group that names them and following only that group’s rules.

The file sits at yourdomain.com/robots.txt, and each subdomain needs its own. A group starts with one or more User-agent lines, followed by Disallow and Allow lines for paths. A group for * covers every crawler that has no group of its own.

The rule that trips people up is that a crawler with its own group ignores the * group completely (RFC 9309, section 2.2.1). Some bots also borrow another bot’s rules. Apple says that if a file doesn’t mention Applebot but does mention Googlebot, “the Apple robot will follow Googlebot instructions.”

Take a file with three groups. The * group disallows /cart/, a GPTBot group disallows /drafts/, and a Googlebot group disallows /search/.

								User-agent: *
Disallow: /cart/
							

								User-agent: GPTBot
Disallow: /drafts/
							

								User-agent: Googlebot
Disallow: /search/
							

When an Allow and a Disallow both match a page, the longer path wins, and Google uses the least restrictive rule when they tie (Google’s robots.txt spec). With Disallow: /blog/ and Allow: /blog/public/, a bot can read /blog/public/ and nothing else under /blog/. Bot names aren’t case-sensitive, and paths are.

Changes take a while to land. The robots.txt standard, RFC 9309, lets a crawler keep a saved copy of the file for up to 24 hours. OpenAI says about 24 hours, Perplexity and Meta say up to 24, and Amazon says its bots use a copy up to 30 days old.

A server error can block every bot by accident. If your server errors when a bot asks for the file, bots that follow the standard treat the whole site as off limits. If the file is missing, bots may fetch anything. After any change to your server or host, open yourdomain.com/robots.txt in a browser to confirm it loads.

Check which AI bots your site allows today

Open yourdomain.com/robots.txt and, for each AI bot you care about, write down whether the file allows it, blocks it, or leaves it to the * group. Then check your CDN’s bot settings, because a CDN can block a bot your file allows.

Start with the * group, which every unnamed bot gets, then read each named group. Match each name against the operator’s current page, since copied lists carry names operators no longer document.

Don’t assume someone on your team wrote these rules. A site builder, an SEO plugin, a hosting company or a CDN setting may have added them. Cloudflare changed its default in July 2025 to block AI crawlers (Cloudflare, July 1, 2025). In July 2026 it said that from September 15, new domains would block training and agent traffic by default on pages that show ads, with search left allowed (Cloudflare, July 1, 2026).

The July 2026 post has a warning for Cloudflare customers who block training. A crawler that does both search and training can be blocked for both, so Googlebot, Applebot and BingBot may be blocked for those customers. Check your dashboard to see what’s switched on before you assume your search crawlers are getting through.

To see which bots visit, filter your server logs or CDN dashboard by the names in the table above. The readiness guide covers reading logs for AI traffic.

Block AI crawlers in robots.txt bot by bot

To block AI bots in robots.txt, give each bot its own group with a Disallow rule, copy your private path rules into any group that allows part of the site, and allow the search crawlers you want to find you.

This file blocks the main training bots and tokens and stays in AI search, apart from Gemini app answers. Amazon’s training opt-out is the page tag covered earlier, so it isn’t in this file. Disallow: / blocks the whole site, because every path starts with a slash. Check it against each operator’s page before you copy it, because names change.

# OpenAI training
User-agent: GPTBot
Disallow: /

# Anthropic training
User-agent: ClaudeBot
Disallow: /

# Common Crawl archive
User-agent: CCBot
Disallow: /

# Meta training. Meta also uses this bot for some product indexing.
User-agent: meta-externalagent
Disallow: /

# Google training token. Also removes this site from Gemini app answers.
User-agent: Google-Extended
Disallow: /

# Apple training token
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

If part of your site is private, repeat that path in every named group that allows the rest of the site, because a bot with its own group won’t read the * group. A group with Disallow: / already covers it. The standard warns that listing a path “exposes” it publicly, so a path whose only protection is that nobody knows it exists should stay out of the file.

If a site builder or SEO plugin writes your file, make the change in that tool’s settings. Then open the live file in a browser and confirm it matches what you wrote.

Robots.txt can’t undo past AI crawling

Robots.txt can’t remove content from models that were already trained, and it can’t stop a bot that ignores it. It also can’t keep a page secret, because anyone can read the file and it locks nothing.

Google’s Gary Illyes called robots.txt “an access preference protocol” that “doesn’t dictate what a system does with the data after it has already crawled it,” in remarks covered by Search Engine Journal on October 22, 2026.

Of the operators in this guide, only Google and Apple offer a robots.txt token that governs how content fetched by their main crawler may be used. Google doesn’t say whether blocking Google-Extended covers pages it fetched before the block, and neither company says a token removes content from a model that was already trained.

Your file doesn’t cover copies of your pages. A page quoted on another site, syndicated, or saved in an archive can reach a model through that copy.

The readiness guide explains the honor system behind the file.

Robots.txt won’t take a page out of Google either. Google can list a disallowed URL if other sites link to it, and it says the file “is not a mechanism for keeping a web page out of Google” (Google robots.txt intro). Use a noindex tag, and leave the page unblocked, because Google has to crawl it to read the tag (Google, block indexing).

Anyone can fake an AI bot, so verify it

You check a bot by confirming the request came from an address its operator controls, using the operator’s published list of addresses or a DNS lookup checked in both directions.

Anyone can send a bot’s name. In a podcast interview covered by Search Engine Roundtable on October 30, 2026, Google’s Gary Illyes said AI has made Googlebot impersonation worse, and Common Crawl warns of “crawlers falsely identifying themselves as CCBot.” If you use a CDN such as Cloudflare, it labels the well-known bots as verified for you. The steps here are for reading your own logs.

The simplest check compares the IP address in your logs against the operator’s published list. Fetch the list again on a schedule, because operators update those files.

How do you verify Googlebot with reverse DNS?

For the three operators with a domain in the last column, you can also run a reverse DNS check. DNS links names to addresses, and a hostname is a computer’s name in it. A reverse lookup goes from address to hostname, and a forward lookup goes back. These are Google’s four steps.

1. Look up the hostname for the IP address in your logs.
2. Confirm the hostname ends in googlebot.com, google.com or googleusercontent.com.
3. Look up the IP address for that hostname.
4. Confirm it matches the address in your logs.

Steps 3 and 4 matter because whoever controls an IP address can set its reverse record to any name, so a hostname check alone passes an impostor. Google’s example is host 66.249.66.1, and nslookup does the same on Windows. The readiness guide lists all four ways to verify an AI agent.

Your CDN can stop AI bots that ignore robots.txt

You stop AI bots that ignore robots.txt at your CDN, firewall or host, with rules that block or slow requests by verified identity, IP range or behavior.

The safest rule pairs a bot’s name with the operator’s IP list. The operator’s own bot gets the rule you meant, and an impersonator can’t ride on its name. Avoid a rule that matches any name containing bot or crawler, because it also blocks Googlebot.

Cloudflare’s AI Crawl Control lets a site “set allow or block rules for individual crawlers” and identifies “which crawlers are violating your directives.” Since July 2026 it sorts AI traffic into Search, Agent and Training, which sit close to the three jobs in this guide.

On a hosted site builder with no access to those rules, check the host’s bot or security settings, or ask its support team.

Large operators have been accused of ignoring the file too. In August 2025, Cloudflare reported that Perplexity reached test sites that disallowed all bots using an undeclared user agent and addresses outside its published ranges (Cloudflare, August 4, 2025). Perplexity disputed the report and said Cloudflare had attributed a third-party browser service’s traffic to it (Perplexity, August 2025).

Recheck your AI bot rules every quarter

Recheck your robots.txt file and your CDN’s bot settings against each operator’s page every quarter, because operators add and rename bots.

One change is already on the way. Google said on October 30 that it plans to support a shared “policy token” for AI training in 2027, “along with other AI providers.” It’s based on an IETF draft, a proposed internet standard that may change. The draft writes the token as User-Agent: __usage_ai-training above Disallow: /. A * group never counts for it, so the preference exists only where a site writes it out.

If a site opts out through the token and doesn’t name Google-Extended, Google plans to treat the file as if Google-Extended were disallowed. Until then, keep using the named rules. Google reads Google-Extended for training.

The October 23 and October 30 search news roundups cover these announcements.

Frequently asked questions

What is GPTBot?

GPTBot is OpenAI’s training crawler, and blocking it tells OpenAI not to train on your content. ChatGPT search uses a different bot, OAI-SearchBot.

What is ClaudeBot?

ClaudeBot is Anthropic’s training crawler, and blocking it signals that your future material should be left out of training. Claude’s search uses a different bot, Claude-SearchBot.

What is OAI-SearchBot?

OAI-SearchBot is the crawler behind ChatGPT search. If you want to appear in ChatGPT search answers, allow it.

How do I block GPTBot?

Add a User-agent: GPTBot group with Disallow: / to your robots.txt file. If you only block part of the site, copy any private path rules from your * group into it, because GPTBot will stop reading the * group.

What is Google-Extended?

Google-Extended is a robots.txt token with no crawler behind it. It controls Gemini training and whether Gemini app answers can use your pages, and Google says it has no effect on Search inclusion or ranking.

Does blocking AI crawlers hurt SEO?

Blocking AI training crawlers such as GPTBot, ClaudeBot and CCBot doesn’t affect Google rankings, and neither does Google-Extended. Blocking an AI search crawler such as OAI-SearchBot can keep you out of that company’s answers, and blocking Googlebot stops Google from crawling the site. The readiness guide explains how blocking each provider’s search crawler affects SEO.

Can I keep my site out of AI Overviews without leaving Google Search?

Yes. The Search generative AI setting in Search Console removes a site from AI Overviews, AI Mode and Discover’s AI features, and Google says it doesn’t affect ranking. The site has to be verified in Search Console first.

Do AI crawlers follow robots.txt?

Most AI training and search crawlers, including GPTBot, ClaudeBot and OAI-SearchBot, say they follow robots.txt. Several user-triggered fetchers, including ChatGPT-User, Perplexity-User and Google-Agent, say they may skip it. Any program can fake a bot’s name and ignore the file.

Can I block an AI company from content it already crawled?

A block can’t remove content an AI company already crawled, because it only applies to future visits and can’t reach models that were already trained. Google-Extended also governs grounding in Gemini app answers, and Google doesn’t say whether that covers pages fetched before the block.

Is Your Website Ready for AI Agents? covers AI traffic beyond crawlers, and What Is an AI Agent? covers the systems that act on a page. Questions about your own site are welcome through the contact page.