A website can be public and still ask automated crawlers not to fetch every path.
The mechanism is commonly called the Robots Exclusion Protocol. A site publishes a plain-text file at /robots.txt. Crawlers read the file, match the rules that apply to their user agent, and use those rules to decide which paths they should avoid fetching.
The protocol is standardised in RFC 9309. The important Wintour House distinction is that robots exclusion answers an access question: should this crawler fetch this path? It is not, by itself, authentication, confidentiality, ranking control or a universal guarantee of de-indexing.
This article sits beneath How Web Crawling Works and beside How Crawl Politeness Works. Crawling discovers. Politeness controls request pressure. Robots exclusion communicates path-level crawling preferences.
1. The File Lives at the Root
A robots file is conventionally located at the root of the relevant origin. A crawler requests that file before or during crawling and applies rules to URLs on that origin.
The location matters because the protocol is scoped. A robots file is not a free-floating site policy hidden somewhere inside the content tree.
2. User-Agent Groups Decide Who the Rule Is For
Rules can target particular crawler user agents or use a wildcard group for a broader set of crawlers.
This allows a publisher to communicate different access expectations to different automated agents, although crawler compliance still depends on the agent respecting the protocol.
3. Allow and Disallow Match URL Paths
Rules typically describe paths that should be allowed or disallowed. The crawler compares the requested URL path against the applicable rule set.
The mechanism is syntactic, not semantic. A rule does not understand that one directory is “private family documents” and another is “low-value filters.” It matches paths according to protocol rules.
4. Robots.txt Is Not a Security Boundary
This is the most important misconception.
A robots file is publicly readable. It can reveal the existence of paths. A crawler can ignore it. A human can type the URL directly. Sensitive information should therefore be protected by authentication, authorisation and proper access control rather than by a robots rule.
The existing Permission-Aware Indexing branch owns the deeper problem of keeping private content out of search systems without relying on public hints.
5. Blocking Crawl and Blocking Indexing Are Different
If a crawler is prevented from fetching a page, it may not see page-level directives contained inside that page.
Search systems can also discover a URL from links without fetching the page content. That means “not crawled” and “not known” are not identical states.
Publishers should therefore choose controls according to the actual goal: reduce crawling, prevent indexation, protect private content, consolidate duplicates or retire a URL. These are different mechanisms.
6. Robots Rules Can Save Crawl Attention
Large sites can generate search-result pages, filters, session paths, temporary states and other low-value URL spaces. Excluding appropriate crawl paths can help reduce crawler effort spent on states that do not add useful searchable content.
This should be used carefully. Blocking a path also prevents the crawler from seeing links and content inside it through that route.
7. Robots and Sitemaps Can Coexist
A robots file can also point crawlers toward XML sitemaps. The two mechanisms solve different parts of discovery: robots communicates access rules; sitemaps expose a structured list of URLs the publisher wants crawlers to know about.
8. Worked Example: Faceted Navigation
An online catalogue can generate thousands of combinations such as colour, size, sort order and price range.
If most combinations contain no unique search value, the publisher may decide to restrict some crawl paths while preserving the canonical category pages that users and search systems should discover.
The decision should be architectural, not cosmetic. The goal is to keep the searchable resource graph coherent.
9. Worked Example: Admin Paths
Administrative interfaces generally should not be exposed to public search. Robots rules can reduce compliant crawler traffic, but real protection still comes from authentication, permissions and application security.
This is a useful example of layered control: public crawler instruction outside, real access enforcement underneath.
10. A Robots-Exclusion Checklist
- Identify the real objective: crawl control, indexing control or security.
- Use robots only for the crawl-control part.
- Place the file at the correct root.
- Group rules by crawler identity where needed.
- Test path matching carefully.
- Avoid blocking resources necessary for understanding important public pages without a reason.
- Use authentication for sensitive content.
- Coordinate robots policy with canonicalisation and sitemap design.
- Review rules after major site-architecture changes.
11. Read the Mechanism Forward, Backward and Sideways
Forward: crawler identifies itself → fetches robots policy → matches rules → allows or suppresses path fetch. Backward: start from an unexpected crawling gap and ask whether a path rule blocked the acquisition layer. Sideways: compare crawler, publisher, security engineer and search user. Each interprets “blocked” differently unless the control layer is named precisely.
Robots exclusion is a sign on the crawler entrance, not a lock on the building.
Continue through How Web Crawling Works, How Indexing Works and the How X Works hub. Next: XML sitemaps — how publishers offer a structured discovery list without pretending the list guarantees indexing.