How to allow AI crawlers in robots.txt so ChatGPT, Perplexity and Google AI Overviews can find and cite your Uttarakhand business website.
To allow AI crawlers in robots.txt, make sure your file does not block Googlebot, OAI-SearchBot (ChatGPT search) or PerplexityBot. Training bots such as GPTBot and Google-Extended are separate choices: blocking them does not remove you from Google Search, and allowing search bots is what lets AI answers cite your pages.
Many small business websites in Ramnagar, Haldwani and across Uttarakhand were built years ago from templates, plugins or hosting panels that added robots.txt rules nobody has looked at since. Some security and CDN settings now block AI bots by default, so a resort or clinic can be invisible to ChatGPT without the owner knowing.
This guide explains which bots matter, which ones you can safely refuse, and how to check your own file in about ten minutes.
What is robots.txt and why does it matter for AI search?
A robots.txt file is a plain text file at yourdomain.com/robots.txt that tells crawlers which URLs they may access, according to Google Search Central. It is mainly used to avoid overloading a site with requests.
Google also notes that robots.txt is not a way to keep a page out of Google; for that you need noindex or a password. Reputable crawlers obey it, but the same page warns that not every crawler does.
For AI search, the file matters because answer engines can only cite pages their crawlers are allowed to read. One old Disallow line can quietly shut your business out of ChatGPT or Perplexity answers.
Which AI crawlers should a local business allow?
The key idea is that each AI company runs different bots for different jobs. Some fetch pages for search answers; others collect content for model training. You can treat them separately.
- Googlebot: Google says pages must be indexed and eligible to show with a snippet to appear as links in AI Overviews and AI Mode. Blocking Googlebot blocks both normal Search and these AI features.
- OAI-SearchBot: OpenAI's crawler documentation says this bot is used to surface websites in ChatGPT search. If it is disallowed, your pages are not shown in ChatGPT search answers.
- PerplexityBot: Perplexity says it uses this crawler to index the web for its search engine, and that it can be blocked with a normal robots.txt record.
- GPTBot: OpenAI says this bot collects content that may be used to train its models. Its documentation states that the GPTBot and OAI-SearchBot settings are independent of each other.
- Google-Extended: Google describes this as a robots.txt token that controls whether crawled content may be used to train future Gemini models. Google states it does not affect inclusion or ranking in Google Search.
How to allow AI crawlers in robots.txt: step by step
You do not need a developer for the check itself; editing the file may need access to your hosting panel or website builder.
- Open yourdomain.com/robots.txt in a browser. If the page is empty or missing, crawlers are allowed by default.
- Look for a line reading User-agent: * followed by Disallow: /. This blocks every bot, including Googlebot, and is the most serious mistake to fix.
- Search the file for Googlebot, OAI-SearchBot, PerplexityBot and Bingbot. Remove any Disallow: / rule listed under these names.
- Decide on training bots. If you prefer your content not to be used for training, add User-agent: GPTBot with Disallow: / and User-agent: Google-Extended with Disallow: /. This is a business choice, not an SEO requirement.
- Keep genuinely private paths blocked, for example admin, cart or login pages, using path-level Disallow rules instead of blocking the whole site.
- Check your CDN or firewall settings as well (see the next section), because they can block bots before robots.txt is even read.
- Use the URL Inspection tool in Google Search Console to confirm Google can fetch your key pages, then re-check the file after any website redesign or plugin update.
Can Cloudflare or my hosting block AI bots without me knowing?
Yes, it is possible. In July 2025 CNBC reported that Cloudflare would ask every new domain whether to allow AI crawlers, with blocking presented as the default. This matters if your website uses Cloudflare or a similar CDN or security service.
These settings work at the network level, so a clean robots.txt is not enough if the CDN refuses the bot. Log in to your CDN or hosting dashboard, look for bot or AI crawler settings, and make sure search-related bots are permitted.
Security plugins on WordPress can do the same. If you are unsure what a setting does, ask your hosting provider before changing it.
Do I need special markup or an llms.txt file to appear in AI Overviews?
Google says there are no additional technical requirements for AI Overviews or AI Mode beyond being indexed and snippet-eligible, and no special schema or AI text files are needed. We covered this in more detail in our post on whether you need llms.txt.
If you want to limit what Google shows from a page, Google points to snippet controls such as nosnippet and max-snippet rather than blocking Googlebot. The same Google guidance notes that recrawling after a change can take days to months.
So the order of work is simple: let the right bots in, keep pages fast and indexable, and then focus on clear, useful content that answers customer questions. Our earlier guide on getting cited by ChatGPT and AI Overviews covers the content side.
How HireRekrut helps
HireRekrut is a digital marketing agency and software company in Ramnagar, Uttarakhand, working with businesses since 2018. As part of our SEO and AI search visibility services, we can review your robots.txt, CDN bot settings and Search Console access, explain what each rule does, and help you decide which AI bots to allow. Call +91 90274 84435 or visit our office at Jaspuria Line, Ramnagar.
FAQs
Will blocking GPTBot remove my website from ChatGPT search?
Not by itself. OpenAI's crawler documentation treats GPTBot (training) and OAI-SearchBot (search) as independent settings, so you can block one and allow the other.
Does blocking Google-Extended hurt my Google rankings?
Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It only controls use of content for Gemini training and grounding, as described in Google's crawler list.
What happens if my website has no robots.txt file?
Without a robots.txt file, crawlers assume they may access the whole site. That is usually fine for a small business, though a file lets you keep admin or login paths out of crawling.
Is robots.txt enough to stop all bots?
No. Google notes that robots.txt rules depend on the crawler choosing to obey them, and some crawlers may not. Use passwords or noindex for anything truly private.
How do I allow AI crawlers in robots.txt on a website builder?
Most builders and WordPress SEO plugins have a robots.txt editor in their settings. Remove any Disallow: / rule for Googlebot, OAI-SearchBot or PerplexityBot, save, and open yourdomain.com/robots.txt to confirm the change is live.
Sources
- Overview of OpenAI Crawlers — OpenAI
- PerplexityBot — Perplexity
- Google's common crawlers — Google for Developers
- AI features and your website — Google Search Central
- Introduction to robots.txt — Google Search Central
- Web giant Cloudflare to block AI bots from scraping content by default — CNBC (via NBC New York)