When ChatGPT, Claude, or Perplexity wants to answer a question by browsing the web, the first thing each tool does is request your site's robots.txt file. That file tells crawlers what they are allowed to read. If your robots.txt blocks the major AI crawlers, your site becomes ineligible to be fetched and cited by those tools when they answer real-time queries, and (separately) excluded from the training data future versions of those models are built on. The exact effect varies per crawler, but the cumulative result for a small business is meaningfully reduced visibility in AI search. We have audited dozens of small business sites in the past year, and a meaningful percentage have this block in place without the owner ever knowing they enabled it. Here is how to check, and how to fix it if you find it.
How to check your robots.txt in 60 seconds
Open a new browser tab and type your domain followed by /robots.txt. For example: https://yoursite.com/robots.txt. You should see a small text file. If you get a 404 or a blank page, that is fine: it means no crawlers are blocked. If you see content, scan it for lines that start with User-agent followed by any of these names: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, CCBot, Bytespider, FacebookBot, Applebot-Extended. If any of those user-agents are followed by a Disallow: / line, that crawler is blocked. If your site runs behind Cloudflare, read the top of the file too. Cloudflare can add its own lines there, between a # BEGIN Cloudflare Bot Preference Sync line (or # BEGIN Cloudflare Managed content on older setups) and a matching # END line, sometimes with a Content-signal line such as ai-train=no. Those lines come from your Cloudflare settings, not from your CMS or SEO plugin, so they only change when you change the setting in Cloudflare.
What each crawler does
OpenAI runs three. GPTBot collects training data. OAI-SearchBot decides whether your pages can show up in ChatGPT search. ChatGPT-User fetches a page when someone in a chat asks ChatGPT to open it, and OpenAI notes that robots.txt may not apply to those user-initiated visits. Anthropic splits the same way: ClaudeBot for training, Claude-SearchBot for search, and Claude-User for pages a person asks Claude to read (anthropic-ai is an older name). Google-Extended is not a separate crawler: it is a robots.txt token that controls whether Gemini can train on your content and use it to ground live Gemini answers. It does not affect Google Search, including AI Overviews, which run on regular Googlebot crawling. PerplexityBot is Perplexity's crawler. CCBot is Common Crawl, which feeds many of these systems indirectly. Blocking any one of these closes a specific channel.
Why your site might be blocking these without you knowing
Three common ways this happens. First, Cloudflare. Its AI bot settings are easy to switch on without thinking about search, and since September 15, 2026, new domains added to Cloudflare start with AI training and agent bots blocked on pages that show ads. Cloudflare can also write its own lines into your robots.txt (covered below). Second, your CMS or host. Some WordPress security plugins and some managed hosts (especially WP Engine and Kinsta variants) added default AI blocking after public discussion in 2023 and 2024. Third, your developer or agency may have added these blocks during a privacy or content-protection sweep without flagging it as a search trade-off.
How to fix it: WordPress
If you are on WordPress and the block is plugin-driven, look in your SEO plugin (RankMath, Yoast, or All in One SEO) for a robots.txt editor. Remove the lines blocking AI crawlers and save. If the block is from a security plugin (Wordfence, iThemes Security), check that plugin's crawler settings. If you can't find the source, the simplest fix is to override robots.txt by adding your own version: most SEO plugins allow this directly from their dashboard.
How to fix it: Cloudflare
Cloudflare replaced its single Block AI bots switch in July 2026. AI traffic is now sorted by purpose: Search (crawlers that index your pages for AI search answers), Agent (fetches made in real time on a person's behalf, like ChatGPT opening your page mid-chat), and Training (crawlers collecting data to train models). Each can be allowed, blocked only on pages that show ads, or blocked everywhere. Log into Cloudflare, select your site, and find these in its Security settings with the AI bot controls (Cloudflare's AI Crawl Control). For most small businesses the practical setup is Search and Agent allowed; Training is your call. Two things to check while you are there. First, a Training block reaches further than the name suggests: since September 15, 2026, crawlers that do both search and training are held to your Training setting, and Cloudflare names Googlebot, Applebot, and Bingbot among them, unless the crawler meets Cloudflare's transparency terms. That includes the old Block AI bots switch, so if you turned it on at some point, confirm Google can still reach your pages. Second, Bot Preference Sync, which is on by default for new Cloudflare customers and available on every plan, writes these choices at the top of your robots.txt, including a no-training line when Training is blocked. To change those lines, change the setting in Cloudflare. And a clean robots.txt does not rule Cloudflare out: its AI bot settings stop the requests themselves, which robots.txt never shows.
Cloudflare's Search, Agent, and Training categories, the September 15, 2026 defaults, and how multi-purpose crawlers are handled.
How Cloudflare writes your AI bot settings into robots.txt, and the markers it uses.
How to fix it: Shopify, Wix, Squarespace
On Shopify, robots.txt was historically locked but is now editable through the robots.txt.liquid template (Online Store > Themes > Edit code). On Wix and Squarespace, robots.txt is auto-generated and not directly editable. Both platforms allow AI crawlers by default, so the most likely cause of blocking on these platforms is a Cloudflare or third-party CDN sitting in front of your site. Check there first.
What good looks like after the fix
After you remove the blocks, fetch your robots.txt again to confirm. The minimum viable version for AI visibility allows all the major AI crawlers (no Disallow lines for them). The cleanest version explicitly allows them: User-agent: GPTBot followed by Allow: /. This is functionally the same as no entry at all, but it serves as visible documentation that you have made an intentional choice. Most major sites that want to be cited in AI now publish a robots.txt that explicitly lists the AI crawlers they allow.
How long until AI tools start seeing my site again?
AI crawlers typically re-fetch robots.txt within 24 to 48 hours. After that, your pages become eligible to be fetched and used in AI answers when relevant queries come in. Re-citation from AI tools follows the same indexing latency as Google: you will not see immediate inclusion, but within a few weeks of being unblocked, your business should start appearing in answers to relevant queries that you would expect to be cited for.
What this fix does not do
Allowing AI crawlers makes your site eligible to be cited. It does not guarantee you will be, and there are specific reasons AI still might not mention your business. The factors that determine actual citation are largely the same as traditional SEO factors plus structured data (schema markup), clear topic focus, and authoritative content depth. For local businesses specifically, your Google Business Profile drives more local AI and search traffic than any single change to your website. We covered the broader landscape in our 16 SEO truths post.
The full set of small-business SEO and AI-search recommendations this post is excerpted from.
Companion guide for local businesses. Google Business Profile drives more local search and AI traffic than your website does.
How Stackra measures AI citation readiness and what signals matter most for appearing in ChatGPT, Perplexity, and Google AI Overviews.
A plain-English explanation of GEO: what it is, why it matters for small businesses, and how it differs from traditional SEO.
Stackra checks your robots.txt, schema markup, and AI-readiness as part of every scan.
Frequently asked questions
Common questions about robots.txt and AI crawlers.
Should small businesses block AI crawlers?
For most small businesses, no. Blocking AI crawlers makes your business invisible to a fast-growing share of search traffic without giving you anything meaningful in return. The privacy and copyright reasons cited for blocking apply more to media and publishing companies whose business model is selling content. A typical local service business or SMB has the opposite problem: not enough discoverability, not too much.
Will allowing AI crawlers slow down my site?
No, in any meaningful sense. AI crawlers respect rate limits and crawl politely. The total bandwidth from AI crawlers across a typical small business site is a tiny fraction of normal traffic. If you do see suspicious load, the right response is per-crawler rate limiting, not a blanket block.
Is GPTBot the same as ChatGPT?
Not exactly. GPTBot is OpenAI's training crawler. OAI-SearchBot decides whether your pages can appear in ChatGPT search, and ChatGPT-User is the fetch when someone in a chat asks ChatGPT to open a specific page. Blocking GPTBot while allowing OAI-SearchBot and ChatGPT-User means you opt out of training but can still show up in ChatGPT's answers. Some sites use this configuration deliberately.
What about Common Crawl (CCBot)?
Common Crawl is a non-profit web crawler whose data feeds many AI training datasets indirectly. Blocking CCBot does not block AI tools at query time, but it removes your site from the training corpora that future LLM versions are built on. For most small businesses, this is a meaningful long-term visibility loss with little upside.
Where is robots.txt actually stored on my server?
Physically, robots.txt is just a text file in the root directory of your web server. On WordPress and most modern CMS platforms, it is generated dynamically and may not exist as an actual file: the platform serves the response when the URL is requested. On a static site or a custom-built site, it lives at /public/robots.txt or /static/robots.txt. To edit it, you typically use your CMS interface or your host's file manager rather than editing the file directly.