Robots.txt is a plain text file in your site's root directory that tells crawlers which URLs they may and may not request. Every crawler that respects the rules checks it before touching your site, and since 2022 the format is an actual internet standard, RFC 9309. For twenty years this file mostly managed Googlebot's crawl budget. In 2026 it has a second job that matters just as much: deciding which AI systems get to read your content at all.
The short version
- Robots.txt lives at your-domain.com/robots.txt and controls crawler access.
- It blocks crawling, not indexing. Blocked pages can still show in results.
- It is a request, not security. Compliant bots obey; hostile ones do not.
- AI crawlers like GPTBot now check it too, and blocking them is a business decision.
The directives that matter
# Allow everything, point crawlers at the sitemap
User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xml
# Keep bots out of admin, allow one file inside it
User-agent: *
Disallow: /admin/
Allow: /admin/public-tools.html- User-agent names the crawler a rule block applies to. An asterisk matches all of them.
- Disallow forbids a path. An empty value forbids nothing.
- Allow carves an exception inside a disallowed path.
- Sitemap points crawlers at your XML sitemap, full URL required.
- Wildcards work in paths. Disallow: /*.pdf blocks every PDF, and the dollar sign pins a match to the end of a URL.
Google reads at most 500KB of the file, and a missing robots.txt is treated the same as one that allows everything.
The AI crawler decision every site now faces
A wave of new user-agents arrived with the AI boom, and they do different things. Training crawlers like OpenAI's GPTBot, Anthropic's ClaudeBot, and Common Crawl's CCBot collect content that may train future models. Retrieval bots fetch pages live so an AI engine can cite them in an answer. Google-Extended is stranger still: blocking it keeps your content out of Gemini training while leaving normal Google Search untouched.
Blocking them all feels safe and costs visibility. AI engines increasingly sit between your buyers and your website, and a bot that cannot read your pages cannot cite your business. Our position, and the setting on this site, is to allow the crawlers and compete for the citations, the same logic behind publishing an llms.txt file that tells AI systems what the site is. A publisher whose content is the product can reasonably choose the opposite. The point is to choose, not to inherit a default.
The mistakes that take sites out of Google
- Disallow: / left over from staging. One character blocks the whole site, and it happens after relaunches constantly.
- Blocking CSS and JavaScript. Google renders pages. If it cannot fetch your styles and scripts, it cannot see the page you built.
- Using robots.txt to hide private pages. The file is public and readable by anyone, so it doubles as a map of everything you consider sensitive. Use authentication instead.
- Blocking a page to deindex it. Blocked pages cannot be crawled, so Google never sees the noindex tag you added. Unblock, let the tag be read, then decide.
- Test before shipping. Search Console's robots.txt report shows exactly how Google parses your file and which rule blocks a given URL.
What a healthy 2026 file looks like
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# AI crawlers allowed deliberately (visibility over blocking)
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://www.example.com/sitemap.xmlShort, deliberate, and current. The Sitemap line earns its keep because it hands every crawler the list of URLs you actually want found, which is the job of the sitemap itself. If an audit finds your file blocking rendering assets, carrying Crawl-delay lines from 2014, or missing entirely, that cleanup belongs to the same pass as the rest of your technical SEO.
Questions people ask about robots.txt
Where does robots.txt go?
In the root of the domain, reachable at your-domain.com/robots.txt, named exactly robots.txt in lowercase. Each subdomain needs its own file; blog.example.com does not inherit the one on www.
Does robots.txt stop a page from being indexed?
Not reliably. It stops crawling, not indexing. A blocked URL can still appear in results if other sites link to it, just without a description. To keep a page out of the index, let it be crawled and give it a noindex tag.
Should I block AI crawlers in robots.txt?
It depends on what you sell. Blocking training bots like GPTBot keeps your content out of future model training but also out of the AI answers your buyers increasingly read. For most businesses that want customers, being cited by AI engines is worth more than the content protection.
Is Crawl-delay still worth setting?
Google ignores it entirely and Bing has moved rate controls into Bing Webmaster Tools. Leaving Crawl-delay in a modern file mostly signals the file has not been reviewed in years.
What does robots.txt do that a sitemap does not?
They are opposites. Robots.txt tells crawlers where not to go, while the sitemap lists the URLs you want found. The Sitemap line inside robots.txt is what connects the two, pointing every crawler at that list.
