Robots.txt is a plain text file at the root of your domain that tells crawlers which URLs they may request. Every compliant bot, from Googlebot to GPTBot, checks it before touching anything else on your site. Since September 2022 the format is a real internet standard, RFC 9309, which settled decades of ambiguity about how the file gets parsed. For most of its life robots.txt existed to steer search engine crawl budget. In 2026 it carries a second decision of equal weight: which AI systems may read your content, train on it, and cite your business in answers. This page covers the whole file, the exact syntax and precedence rules, the crawling versus indexing distinction, the AI crawler roster, testing, and the mistakes that quietly take sites out of Google.
The short version
- Robots.txt lives at your-domain.com/robots.txt and controls crawler access per protocol and host.
- It blocks crawling, not indexing. A blocked page can still rank, badly, off its links alone.
- The most specific user-agent group applies, and the longest matching path rule wins.
- AI crawlers split into training, search-index, and user-fetch bots. Decide per job, not per vendor.
- It is a request, not security. Compliant bots obey; hostile ones read it as a map.
What the file promises and what it does not
RFC 9309 defines robots.txt as a set of access requests that crawlers agree to honor. Nothing enforces it. Googlebot, Bingbot, and the major AI crawlers comply; scrapers and attack tools do not, and the standard never claimed they would. That framing explains everything else about the file: it is a coordination protocol between you and well-behaved software, and its power ends exactly there.
The HTTP status of the file itself changes crawler behavior more than most people expect. Google treats a missing file, or any 4xx response except 429, as if no robots.txt exists, so everything may be crawled. A 5xx or 429 response is the dangerous one: Google reads a server error as a temporary signal that the whole site is disallowed and keeps retrying until it gets a real answer. A misconfigured server that throws 500s on /robots.txt can stall crawling of a healthy site. Google also caches the file for about 24 hours, longer if your Cache-Control max-age says so, which is why a fix never registers instantly. One more hard limit: Google reads only the first 500 KiB. Real files come nowhere near that unless something is generating rules programmatically, at which point the excess is silently ignored.
Every protocol and host needs its own file
A robots.txt file governs exactly one origin: one protocol, one host, one port. The file at https://www.example.com/robots.txt says nothing about blog.example.com, nothing about shop.example.com, and nothing about the http version of the same host. Each of those origins needs its own file at its own root. This bites hardest on platforms that hand out subdomains, staging environments living at dev.example.com, and CDN or media hosts serving your images. If a subdomain should not be crawled, its own robots.txt has to say so, and the file must sit at the root. A robots.txt inside a subdirectory is just a text file nobody reads.
The syntax, one annotated file
# Lines starting with a hash are comments.
# A group begins with one or more User-agent lines
# and holds the rules that follow them.
# Group 1: every crawler that matches nothing more specific.
User-agent: *
Disallow: /cart/
Disallow: /search/
# Allow carves an exception inside a disallowed path.
Allow: /search/how-it-works/
# Wildcard: block every PDF anywhere on the site.
# The dollar sign pins the match to the end of the URL.
Disallow: /*.pdf$
# Group 2: image bot only. Because this group matches
# Googlebot-Image more specifically, it REPLACES group 1
# for that bot. Rules do not combine across groups.
User-agent: Googlebot-Image
Disallow: /drafts/
# Sitemap stands outside every group, full URL required.
Sitemap: https://www.example.com/sitemap.xml- User-agent names the crawler a group applies to. An asterisk matches any bot that no other group names.
- Disallow forbids every path starting with the value. An empty value forbids nothing.
- Allow re-opens a path inside a disallowed one. Google and Bing support it; it is in RFC 9309.
- Sitemap points crawlers at your XML sitemap. It takes a full absolute URL and can appear anywhere in the file.
- * matches any sequence of characters inside a path, and $ anchors a rule to the end of the URL.
Which group a crawler obeys
A crawler picks the single group whose User-agent line matches it most specifically and ignores every other group. This is the rule people miss. If your file has a general block under User-agent: * and then a group for Googlebot that only disallows /tmp/, Googlebot obeys only the /tmp/ rule. The general rules do not apply to it anymore. Any bot you name gets its own complete rule set, so repeat the shared rules inside every named group that should follow them.
Which rule wins inside a group
When several rules match one URL, the most specific rule, meaning the one with the longest matching path, wins. Allow: /admin/tools/ beats Disallow: /admin/ for anything under /admin/tools/ because it is longer. When an Allow and a Disallow match with equal specificity, Google lets the Allow win. Paths are case sensitive, so Disallow: /Private/ does not block /private/, a detail that has burned many migrations off case-insensitive servers.
Blocking a crawl is not deindexing a page
Robots.txt controls whether a bot may fetch a URL. Indexing is a separate decision Google makes about whether the URL appears in results, and a URL can be indexed without ever being fetched. If enough external links point at a blocked page, Google indexes the bare URL with no description, the state Search Console labels indexed though blocked by robots.txt. This produces the classic self-defeating pattern: someone wants a page out of Google, blocks it in robots.txt, and adds a noindex tag. The block means Google can never fetch the page, so it never sees the tag, and the URL lingers in the index. The correct order is the reverse. Leave the page crawlable, serve a noindex meta tag or X-Robots-Tag header, wait for the page to drop out, and only then block it if you also want the crawl budget back.
The AI crawlers and what each one feeds
The AI companies now run separate bots for separate jobs, and the split is what makes a smart robots.txt possible. Training crawlers collect content that may shape future model weights. Search-index crawlers build the retrieval layer that AI answers cite. User-fetch agents grab one page live because a person asked about it, which is why OpenAI and Perplexity both say robots.txt rules may not bind those fetches, while Anthropic states all three of its bots honor the file. Google-Extended is the odd one out: it is not a crawler at all but a product token that Googlebot checks, and blocking it keeps your content out of Gemini training without touching your Google Search presence.
| User-agent | Run by | What it feeds | Blocking it means |
|---|---|---|---|
| GPTBot | OpenAI | Model training | Your content stays out of future GPT training |
| OAI-SearchBot | OpenAI | ChatGPT search index | ChatGPT search stops citing your pages |
| ChatGPT-User | OpenAI | Live fetches for a user | Little; user-initiated fetches may proceed anyway |
| ClaudeBot | Anthropic | Model training | Your content stays out of Claude training |
| Claude-SearchBot | Anthropic | Claude search index | Claude search results lose your pages |
| Claude-User | Anthropic | Live fetches for a user | Claude cannot open your pages when users ask |
| PerplexityBot | Perplexity | Answer-engine index | Perplexity answers stop citing you |
| Google-Extended | Gemini training and grounding | No Gemini use of your content; Search unaffected | |
| CCBot | Common Crawl | Open dataset many models train on | Removes you from a source many labs draw from |
Our position, and the setting on this site: allow the search-index and user-fetch bots even if you block training. AI engines sit between a growing share of buyers and your website, and a bot that cannot read your pages cannot recommend your business. A publisher whose content is the product can reasonably block everything. The failure mode we keep seeing is neither choice but the default, usually a copied blocklist that a developer pasted in 2024 and nobody has read since. Decide per bot, write the decision down, and revisit it yearly, because this roster changes.
The mistakes we keep correcting in audits
- Disallow: / left over from staging. One character blocks the whole site, and it survives relaunches constantly because the staging file ships with the code. Put a robots.txt check in the launch checklist, not in someone's memory.
- Blocking CSS and JavaScript. Old WordPress files still disallow /wp-includes/ or /assets/. Google renders pages, and a page whose styles and scripts are blocked gets judged on the broken version. Unblock everything the page needs to draw itself.
- Missing the trailing slash. Disallow: /blog blocks /blog/, /blog-news/, and /blogging-tips.html, because rules match by prefix. Disallow: /blog/ blocks only the directory. The shorter rule almost always blocks more than intended.
- Blocking every URL with parameters. Disallow: /*?* looks tidy and can cut off paginated categories, tracked landing pages, and faceted navigation you wanted crawled. Block specific parameters you can name, not the whole question mark.
- Naming a bot and forgetting the shared rules. Adding a User-agent: Googlebot group means Googlebot now ignores the * group entirely. Copy the shared rules into every named group.
- Short, deliberate, current. The healthiest files we see fit on one screen, carry a comment explaining each block, and end with a Sitemap line.
How to test the file before and after shipping
Search Console replaced its old standalone tester with the robots.txt report, under Settings. It shows the exact file Google fetched for each host of your property, the fetch time, every parse error and warning, and a button to request an emergency re-fetch after you fix something bad. Pair it with the URL Inspection tool, which names the specific rule blocking any URL you test. Two curl commands cover the rest:
# Confirm the file resolves with a 200, not a redirect chain or 5xx.
curl -sI https://www.example.com/robots.txt
# Read exactly what crawlers receive, straight from the server.
curl -s https://www.example.com/robots.txtRun both against every subdomain, not just www. After any change, remember the cache: Google can keep acting on the old file for a day. For AI bots, your server access logs are the ground truth. Grep them for the user-agent strings in the table above and you will know who actually visits and whether your rules changed their behavior.
Robots.txt is not a security control
The file is public by design. Anyone, including every attacker, can read yours right now, which turns a line like Disallow: /internal/ into a signpost for exactly the content you wanted hidden. Bots that intend harm skip the file or use it as a target list. Anything genuinely private belongs behind authentication, and pages that must stay out of Google need noindex, not a robots rule. Treat robots.txt as traffic direction for polite crawlers and nothing more.
Where sitemaps and llms.txt pick up
Robots.txt is the deny list; the sitemap is the allow list, the set of URLs you actively want found, and the Sitemap line inside robots.txt is the handshake between the two. llms.txt is the newest neighbor: it does not control access at all, it describes your site to AI systems in plain language so the ones you let in understand what they are reading. Together the three files answer three different questions, where bots may go, what you want surfaced, and what it all means. If an audit finds your robots.txt blocking rendering assets, carrying Crawl-delay lines from 2014, or throwing 5xx errors, that cleanup belongs in the same pass as the rest of your technical SEO.
Questions people ask about robots.txt
Where does robots.txt go?
In the root of the host, reachable at your-domain.com/robots.txt, named exactly robots.txt in lowercase. Each subdomain and each protocol needs its own file; blog.example.com does not inherit the one on www, and an http file does not cover https.
Does robots.txt stop a page from being indexed?
Not reliably. It stops crawling, not indexing. A blocked URL can still appear in results if other sites link to it, just without a description. To keep a page out of the index, let it be crawled and give it a noindex tag.
What happens if my site has no robots.txt at all?
Nothing bad. Google treats a missing file, or any 404, as permission to crawl everything. A small site with no areas to protect loses nothing by having no file, though adding one with a Sitemap line is an easy win.
Should I block AI crawlers in robots.txt?
Split the decision by job. Training bots like GPTBot and CCBot feed future models; search bots like OAI-SearchBot and Claude-SearchBot decide whether AI answers can cite you. Most businesses that want customers should allow the search bots even if they block training, because a bot that cannot read your pages cannot recommend your business.
Is Crawl-delay still worth setting?
Google ignores it entirely and Bing prefers the crawl control settings inside Bing Webmaster Tools. Leaving Crawl-delay in a modern file mostly signals the file has not been reviewed in years.
How do I check which rule is blocking a URL?
Open Search Console, go to Settings, then the robots.txt report. It shows the exact file Google last fetched, when, and any parse errors. The URL Inspection tool then tells you whether a specific URL is blocked and by which rule.
