Is your website ready for AI agents?
Updated

On this page
A website is ready for when the systems you allow can find, read, reach, and use it. For an agent, using a site means working the forms and buttons a task needs. Most of that rests on the and accessibility work you already do. The new work is a policy on which systems get in, controls that software can read, and logs that show what those systems do.
What is an AI agent? explains what agents are and how they read and act on a page.
A website is ready for AI agents when five layers hold
A website is ready for AI agents when all five layers hold, from discovery through governance. Discovery and understanding are SEO work, and they cover whether systems can find your pages and tell what they say. Access, interaction, and governance usually sit with infrastructure, front-end, and security teams. A site with solid SEO can fail at any of those three.
Each layer lists a common failure and the team that usually owns the fix.
- DiscoverSEO / content / platform
Can AI systems find the content?
- Crawlability
- Indexation
- Internal links
- Sitemaps
- Feeds
Fails likeAn important page is orphaned.
- UnderstandSEO / content / product
Can they tell what it means?
- Clear writing
- Consistent naming
- Headings
- Structured data
- Accurate facts
Fails likeProduct fitment is unclear or contradictory.
- AccessInfrastructure / security
Can the right system retrieve and render it?
- robots.txt
- HTTP status
- CDN / WAF
- Rendering
- Rate limits
Fails likeA CDN rule returns 403 to intended automated traffic.
- InteractFrontend / accessibility
Can an authorized agent use the site?
- Native links and buttons
- Labeled forms
- Useful errors
- Stable layouts
Fails likeA form control has no programmatic name.
- GovernSecurity / legal / product / search
Are access and permissions intentional?
- Bot policy by purpose
- Verified identity
- Authorization
- Logging
Fails likeTraining and user-triggered access inherit the same blanket policy.
Take a clinic whose service pages are all indexable and written in plain English. An assistant tries to request a visit through the appointment form and fails. Either a security rule refused the request, or the date field has no label software can read. Those are access and interaction failures, and a content audit doesn’t check for either.
Four kinds of AI traffic reach a website
Four kinds of AI traffic reach a website, and each one needs its own access decision. A single rule for every “AI bot” makes all four decisions in one line.
A training collects pages to help build a model. A search crawler collects them so the provider can cite them as sources in its answers. A user-triggered retrieval happens when a person asks an assistant about your page and the provider fetches it for them on the spot. A browser agent opens the page in a browser and works the controls the way a person would. One product can send all four, and ChatGPT has a training crawler, a search crawler, a user-triggered fetcher, and an agent mode.
Each of the four decisions is listed with the systems that belong to it.
Search and discoveryVisibility
Blocking these can remove you as a source in that provider’s search answers.
| System | Operator | Documented control | What blocking changes |
|---|---|---|---|
| OAI-SearchBot | OpenAI | robots.txt: OAI-SearchBot | Removes you from ChatGPT search answers. Your site can appear as a navigational link. |
| Claude-SearchBot | Anthropic | robots.txt: Claude-SearchBot | Can affect whether Claude cites you as a source. |
| Googlebot | robots.txt: Googlebot | Stops Google crawling your pages. Their URLs can stay listed with no description. | |
| PerplexityBot | Perplexity | robots.txt: PerplexityBot | Can remove you as a cited source in Perplexity. |
TrainingContent use
Blocking these is a content-use decision. It doesn't automatically change your search visibility.
| System | Operator | Documented control | What blocking changes |
|---|---|---|---|
| GPTBot | OpenAI | robots.txt: GPTBot | Limits use of your pages for model training. |
| ClaudeBot | Anthropic | robots.txt: ClaudeBot | Limits use of your pages for model training. |
| Google-Extended | robots.txt: Google-Extended (token only) | Opts your content out of Gemini training and grounding without affecting Search. | |
| CCBot | Common Crawl | robots.txt: CCBot | Limits inclusion in Common Crawl, a dataset many models train on. |
| Bytespider | ByteDance | robots.txt: Bytespider (compliance has been inconsistent) | Limits use of your pages for training, if the directive is honored. |
User-triggered retrievalUser request
A person asked an assistant for the page, and blocking the fetch withholds it from them. Some providers document that robots.txt may not apply to these fetches.
| System | Operator | Documented control | What blocking changes |
|---|---|---|---|
| ChatGPT-User | OpenAI | robots.txt may not apply (user-initiated) | Withholds pages a person asks ChatGPT to fetch. Blocking it may take an edge or firewall rule. |
| Claude-User | Anthropic | robots.txt: Claude-User | Affects pages a person asks Claude to fetch. |
| Perplexity-User | Perplexity | Generally ignores robots.txt (user-initiated) | Withholds pages a person asks Perplexity to fetch. Blocking it takes an edge or firewall rule. |
Browser agentsSecurity/interaction
This is a security and interaction decision, and robots.txt doesn't control it.
| System | Operator | Documented control | What blocking changes |
|---|---|---|---|
| Browser-based agents | Various | Bot management, session, and authorization controls | Restricting them affects people who delegate research, forms, or checkout to an assistant. |
Last verified October 2026. Each row reports what the operator documents.
Opting out of OAI-SearchBot is a search visibility decision. OpenAI’s bot documentation says a site that does it won’t be shown in ChatGPT search answers.
An agent’s request takes the same road as a person’s
An agent’s request travels the same road through your infrastructure as a person’s, and it can be stopped before your server ever sees it. That road usually runs through a , then bot management that sorts automated traffic from people, then a , then your own server. Any of them can refuse the request, redirect it, or rate-limit it by capping how many requests one client can make. Your hosting or platform team usually owns the CDN and WAF, and security owns bot management.
Every response comes back with a status code, a three-digit number that says what happened to the request.
A request passes six stops before a response comes back, and any of them can end it. Pick a response to see where it ends and what that code proves.
- AI systemmakes the request
- DNSresolves the host
- CDN / edgenearest cache
- Bot managementclassifies the client
- WAF / securityapplies rules
- Application / originserves the page
- Responsewhat the agent receives
- Render + interactonly a browser session gets this far
403 The request was denied.
- Proves
- This request was refused.
- Doesn't prove
- That every request from the provider is blocked.
- Investigate
- CDN, WAF, and bot-management rules, identity verification, and the request source.
| Response | Stops at | Proves | Doesn't prove | Investigate |
|---|---|---|---|---|
| 200 | Response | This request retrieved content. | That the content is complete, rendered, or usable for the task. | Whether the response contains what the agent needed. |
| 301/302 | Application / origin | The URL moved, and a compliant client can follow it. | That every client follows it, or that the destination returns 200. | The redirect target, chain length, and the status it finally returns. |
| 403 | WAF / security | This request was refused. | That every request from the provider is blocked. | CDN, WAF, and bot-management rules, identity verification, and the request source. |
| 404 | Application / origin | This URL doesn't resolve to content. | That the content is missing from the site. It may live elsewhere. | Internal links, redirects, canonicals, and the correct current URL. |
| 429 | CDN / edge | Request volume from this client crossed a threshold. | That the client is unwelcome, or that the limit is intentional. | Rate-limit policy, traffic volume, and verified-bot handling. |
| 5xx | Application / origin | Something broke on your side for this request. | That the failure is specific to automated traffic. | Application logs and error rates, for this path and overall. |
One status code describes one request. A single 403 from one address means that request was refused, maybe by a WAF rule that fired once. Before anyone reports that a platform is blocked, check whether the refusals repeat across addresses and days, and which layer issued them.
A 200 gets the agent a page, and what it can use from that page depends on how it reads. A lightweight retrieval works with the HTML the server sent. A browser agent goes on to the page and sees what its scripts add. A page can pass one and fail the other.
Plain retrieval sees
- A title and an empty container the script fills later
- No price, no stock status, no product name in the HTML
- An "Add to cart" button that is a div with a click handler
Browser agent sees
- Product name, price, and "In stock" rendered on the page
- Fitment notes and the specification table
- The same div, which it can't identify as a button
Google can render JavaScript for Search and publishes JavaScript SEO guidance. Those practices help agents too, and the unlabeled button fails in both views.
Some AI bots follow robots.txt and some skip it
Whether an AI bot follows depends on the provider and on what the request is for. The file works on the honor system. A crawler that ignores it can request every page on the site. Anything that has to stay private needs a login and access controls.
The answer can differ by purpose inside one company. As of October 2026, OpenAI, Perplexity, Anthropic, and Google documented these positions.
- The crawlers these four providers document for training or search follow the file.
- OpenAI’s bot documentation says robots.txt rules may not apply to ChatGPT-User, because a person triggered the request.
- Perplexity’s crawler documentation says Perplexity-User generally ignores the file for the same reason.
- Anthropic’s crawler page says all of its bots honor it, Claude-User included.
Those positions change, so read the provider’s page before you rely on one.
Google-Extended is a robots.txt token with no crawler behind it. Disallowing it keeps your content out of Gemini training. Google’s crawler documentation says it has no effect on whether you’re included or ranked in Search.
A deliberate robots.txt file ends up with three blocks, one each for training, AI search, and sensitive paths like carts and accounts.
Change a decision to see which lines of the file change with it.
This is a content-use decision, and blocking it has little to do with AI search visibility.
Blocking a documented search crawler can remove you from that provider’s answers.
Carts, accounts, and internal search rarely need to be crawled.
This file is an illustrative example. Verify the current provider token and documented behavior before deploying anything like this.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: *
Disallow: /cart/
Disallow: /account/
Allow: /What this policy means
- Training crawlers are asked to stay out.
- The AI search crawler is allowed, and your pages stay available as sources.
- Every other crawler is kept out of the cart and account paths.
Block AI crawlers by purpose
Block AI crawlers by purpose, because blocking each kind of traffic has a different consequence. The decision for each comes down to identity, risk, and business value.
| Traffic purpose | Why allow it | Why restrict it |
|---|---|---|
| Search and discovery | Visibility, citations, referrals | Abuse or infrastructure concerns |
| User-triggered retrieval | Helps people reach your information through AI | Sensitive or restricted content |
| Browser agents | Can help people research, convert, and complete tasks | Security, account, or transaction risk |
| Model training | Participation in model development | Content-use or licensing policy |
| Unknown automation | Nothing, until you know what it is | Scraping, abuse, cost, security |
Blocking training doesn’t cost you search visibility when the provider separates the two. A publisher that licenses its articles can disallow GPTBot and Google-Extended, allow OAI-SearchBot and Googlebot, and stay in both ChatGPT search answers and Google Search. A parts store that wants to be found would allow search and user-triggered traffic and keep unknown automation away from checkout.
The AI crawlers list gives each bot’s robots.txt name by company and job, with what blocking each one costs.
Each of the four conditions leads to one policy category. Find the row that describes the traffic in front of you.
IfIdentity isn't verified, and the traffic touches sensitive data or carries risk
Require verificationThe consequences depend on who is asking, and the identity is unverified.IfIdentity isn't verified, and there is no visibility benefit
Restrict until understoodIt offers no visibility benefit and carries risk. Restricting it while you work out what it is costs little.IfVerified, but the action reaches private, account, or transactional data
Allow public retrieval, restrict sensitive actionsReading public pages and reaching an account are different decisions. Separate them.IfVerified, supports visibility, and reaches nothing sensitive
Generally allow and monitorAllow it and watch the volume.
This isn't legal advice.
Verify an AI agent with evidence the provider controls
Verify an AI agent with evidence the provider controls, such as a published IP range, a reverse-DNS record, or a signed request. The string can say anything, so a scraper can call itself GPTBot. If your bot rules let named crawlers past the rate limits, a scraper using the name gets past too.
There are four methods. If your site sits behind a CDN with bot management, start with the third, because those checks are already running.
- Published IP ranges list the addresses a provider’s systems use, and you compare the request’s address against the list.
- Reverse DNS looks up the hostname behind the address, and you confirm it belongs to the provider.
- With bot-management verification, your CDN does the work and labels the traffic for you. Cloudflare, for one, classifies bots by identity and behavior.
- Signed requests let automated systems sign each request so the identity can be checked. Cloudflare supports this through Web Bot Auth.
Change the name on the left and nothing else about the request changes. The signature on the right is the part that can be checked.
Identity claim
GET /pricing HTTP/2
User-Agent: GooglebotAnyone can send this request under any name.
Verified identity
GET /pricing HTTP/2
User-Agent: ExampleBot
Signature: keyid="…", sig="…"Web Bot Auth checks the signature against the operator's published key. This identity can be verified.
How verification works
- The request makes a claim about who is sending it.
- Verification checks evidence the provider controls, such as a published key, a published IP range, or a reverse-DNS record.
- Your infrastructure then applies policy based on the verified identity and the observed behavior.
Schema, llms.txt, and WebMCP help only where they are read
Schema, llms.txt, and WebMCP help a website only where a system you care about reads them. None of them makes a site ready for AI agents on its own.
helps any system that supports the types on your page. No universal “AI agent schema” guarantees an agent can use or cite a page. Google has said its generative Search features have no special structured-data requirement.
doesn’t substitute for crawlability, rendering, useful content, accessible controls, or an access policy. Google says it doesn’t use the file for Search or its generative features.
is a community group draft, and it isn’t a W3C standard, a ranking factor, or a prerequisite for basic readiness. I added one WebMCP tool to the contact form on this site as a working example. An assistant can use it to draft a message into the form, and a person has to read the draft and press Send.
The AI agent readiness checklist covers five layers
The AI agent readiness checklist has four items for each of the five layers. Run through it before you change anything on the site.
Work from the top down, because a failure in access makes interaction untestable. A site crawl and answer most of the discovery and understanding items. Access and governance need whoever manages your CDN and security rules, and interaction needs a front-end developer or an accessibility audit. Any item you can’t confirm is a finding to take to that layer’s owner.
Tick what already holds, and the result shows which layers have open items. Your ticks are saved in this browser.
Tick what already holds
Every item is confirmed. The next step is to test one task with an agent.
Test readiness by watching an agent attempt one task
Test AI agent readiness by watching an agent attempt one task from start to finish and noting where it stalls. The checklist points to likely problems, and the task test confirms whether the fixes hold. It adds two stages to the five layers, recovering from an error and reaching the stopping point.
Choose a task a customer would hand to an assistant and that matters to the business. A parts store might pick finding a replacement part and confirming it’s in stock, and a clinic might pick finding the next available appointment. Set the stopping point before anything consequential, so the test ends at the cart or the filled-in form, before a purchase or a submission.
Run the task twice. The first run tests plain retrieval. Paste the page’s URL into an assistant that can browse the web and ask what’s on the page, or read the page source yourself. The second run gives the same task to a browser agent. Note where each attempt stalls and what it could see at that point.
Mark each stage as you watch the test. The form keeps your notes for copying and runs nothing on your site.
- DiscoverCould the agent find the right page?
- UnderstandCould it tell this is the right page for the task, such as the correct product and fit?
- AccessDid the request reach the page, render, and return usable content?
- InteractCould it use the controls the task needs, such as options, a date field, or add to cart?
- RecoverWhen something failed, did the page explain enough to try again?
- Reach the stopping pointDid it get to the right conversion or handoff point?
Test result
Not tested
The point where the task breaks tells you which layer to investigate. If it broke before the page loaded, look at access. If the page loaded and the agent couldn’t tell what a control was for, look at interaction. If something blocked it and nobody on your side can explain why, look at governance. Fix that layer, run the task again, and then choose the next task.
How should you interpret agent-readiness scores?
An agent-readiness score is a diagnostic that covers less ground than the task test. No single score proves an agent can use your website, because each tool tests different conditions. Some check discovery files and bot controls. Others check support for protocols such as WebMCP or MCP, the Model Context Protocol, a standard for connecting AI systems to tools and data. A third kind runs scripted browser and accessibility checks.
Each kind has a blind spot, so an average of their scores tells you nothing about the site. Use a score the way you’d use any audit result, to find conditions worth investigating.
Each kind of test answers a different question.
Standards and access diagnostics
Cloudflare-style readiness audit
Checks
Whether robots.txt and bot controls say what you think they say
Can't tell youWhether anyone could finish a task on the site.
Deterministic browser and interface checks
Browser / Lighthouse-style agentic diagnostics
Checks
Whether every control has a name software can read
Can't tell youWhether the right systems can reach the page.
Outcome test
One task, attempted end to end
Checks
Whether an agent can find the product, confirm the fit, and reach the cart
Can't tell youWhether any other task on the site would get the same result.
What does readiness look like for an ecommerce site?
For an ecommerce site, AI agent readiness also covers product data, the cart, and the account. On a store the task test has a longer path, and there are six more checks.
- Product data is accurate and inventory is current.
- Fitment, meaning which models a part or accessory works with, and specifications are clear enough to act on.
- Product identifiers stay stable from one visit to the next.
- Product and checkout interfaces are usable by software as well as people.
- Account and payment safeguards are in place.
- An API, the defined way one piece of software requests data or actions from another, exists where an integration uses it.
A store can also claim a ChatGPT business profile and upload a product feed, covered in Businesses in ChatGPT and how to add your business.
Measure AI agent traffic in your server, CDN, and WAF logs
Measure AI agent traffic in your server, CDN, and WAF logs, because those record every request software makes to the site. Once the fixes are in, that’s where you see whether anything changed. Analytics records people who clicked through from an AI product, and whether an AI answer cited you is a third measurement. No single tool shows all three.
Each of the three records a different event, in a different place.
- 01
AI search visibility
Were you shown or cited in an AI answer?
Where to look
- Citation tracking
- Answer monitoring
- Search Console (where applicable)
- 02
AI referral traffic
Did a person click through to you from an AI product?
Where to look
- Analytics
- Landing pages
- Conversions
- 03
Automated agent activity
Did software access or use the site?
Where to look
- Server logs
- CDN logs
- WAF data
- Bot-management platforms
One conversation can produce all three. When an assistant’s answer cites your page, that’s AI search visibility. When the person clicks the link, your analytics record the session as AI referral traffic. Before either happened, the provider’s crawler fetched the page, and that fetch shows up in your server or CDN logs as agent activity.
On an inventory program where I tracked citations and referrals, placements in Google’s AI Overviews grew from 4 to 28. AI Overviews are the AI-written summaries at the top of Search results. AI referral traffic to the optimized inventory section grew 352% year over year. The automotive case study has the method, and the technical SEO section of my expertise page covers the access and rendering work underneath it.
Google has added a Generative AI performance report to Search Console for its own experiences, and it’s worth using where your site has it. Everything outside Google’s own experiences comes from the logs.
To read the logs, filter requests by documented crawler names such as OAI-SearchBot or ClaudeBot. That shows which systems arrive, which pages they ask for, and which status codes they get back. Then verify a sample with the methods above before you trust the counts.
Getting content retrieved and cited in AI answers is the work of Answer Engine Optimization (AEO). The Answer Engine Optimization (AEO) guide covers it.
Frequently asked questions
How do I make my website easier for AI agents to use?
The fundamentals that help people, search engines, and assistive technology also make a website easier for AI agents to use. They include clear content, semantic HTML, descriptive controls, labeled forms, stable layouts, reliable rendering, and useful error messages. After those, review bot access, agent policy, and the conversion paths that matter to your business.
Does blocking AI crawlers hurt SEO?
Blocking AI training crawlers such as GPTBot, ClaudeBot, and CCBot doesn’t affect Google rankings, and neither does Google-Extended. Blocking Googlebot stops Google crawling your pages, though Google says a blocked URL can still appear in search results without a description. AI Overviews only link to pages that are indexed and eligible to show a snippet. Blocking a provider’s own search crawler, such as OAI-SearchBot, removes you from that provider’s answers.
Can AI agents access JavaScript websites?
AI agents can often access JavaScript websites, depending on how the agent reaches the site. A lightweight retrieval may see only the source HTML and miss anything added by client-side JavaScript. A browser-based agent renders the page and sees the result, so keep important content and controls from depending on scripts that may never run.
How can I test an AI agent on my website?
To test AI agents on your website, pick one task a customer would hand to an assistant and watch an agent attempt it end to end. Note where it can discover the page, understand the content, use the controls, recover from errors, and reach the goal. Test both a plain retrieval and a browser-based interaction, because a page can work in one and fail in the other.
Are AI agents safe to allow on my website?
Whether AI agents are safe to allow depends on what they can reach. Public retrieval is usually low risk. Anything touching accounts, private data, or transactions belongs behind authentication and authorization, and consequential actions should require a person to confirm.
Should I implement WebMCP?
Implement WebMCP only if a system you care about consumes it and a use case benefits. WebMCP is a community group draft from the Web Machine Learning Community Group. It isn’t a W3C standard, a ranking factor, an AEO requirement, or a prerequisite for basic agent readiness.
How do I know if agents visit my site?
Server logs, CDN logs, WAF data, and bot-management platforms show whether AI agents visit your site, because they record each request made to the server. Analytics shows people who clicked through from AI products. Neither source shows whether an AI answer cited you.