robots.txt
A root file telling crawlers which paths they may fetch — advisory, not enforcement.
robots.txt states crawling rules for well-behaved bots. It is honoured by convention, not enforced: it cannot stop a crawler that ignores it, and it is not a security control. Increasingly it also names AI-specific agents, letting a site admit search crawlers while excluding model-training ones.
Key facts
- robots.txt is advisory: it is not access control, and a path listed there is still reachable by anyone who requests it.
- Disallow prevents crawling, not indexing — a blocked URL can still appear in results from external links, because the crawler cannot read the noindex it was blocked from fetching.
- AI crawlers are commonly addressed by name, so a site can allow search indexing while excluding training crawlers.
- A Sitemap line is valid in robots.txt and is one of the standard ways a crawler discovers a sitemap.
Frequently asked questions
Does Disallow keep a page out of search results?
Not reliably. It stops the page being fetched, which also stops any noindex on it being seen. To keep a page out of results, allow crawling and serve noindex, or require authentication.
How do I block AI training crawlers but keep search traffic?
Name the training agents explicitly in their own groups and leave the search crawlers permitted. The distinction is by user-agent, so it depends on the crawler identifying itself honestly.
Machine-readable copy of this page:
/glossary/robots-txt.md