robots.txt, noindex, and Access Control

Crawl, Index, and Access Are Different Controls

robots.txt lives at the origin root, such as https://example.com/robots.txt. It tells cooperative crawlers which paths they may request. It is public and is not a security boundary.

Disallow controls crawling, not guaranteed indexing. A blocked URL can still be known through links or other signals. If you need a crawlable HTML page kept out of search, use a robots meta noindex directive and allow the crawler to fetch the page so it can see the directive.

For non-HTML resources such as PDFs, an X-Robots-Tag: noindex header can carry index directives. For private or staging content, use authentication, network restrictions, or another real access-control mechanism.

The Order of Operations Matters

A directive only works if the relevant system can see it. This is why the sequence matters:

1. A crawler checks robots.txt before requesting a URL.
2. If crawling is allowed, the crawler can request the page or file.
3. If the response is fetched, page-level meta directives or headers can be read.
4. If indexing is allowed, the page can be considered for the index.

So a public HTML page that should stay out of search often needs to be crawlable with noindex, not blocked. A private dashboard needs a login, not noindex. A deleted page needs an honest response, not a robots rule that hides the evidence.

Use robots.txt mainly to reduce unnecessary crawling of public low-value patterns, not to clean up every index problem.

Decision Table

A Small robots.txt

User-agent: *
Disallow: /internal-search/
Disallow: /thank-you/

Sitemap: https://example.com/sitemap.xml

This file is a crawler instruction, not a content inventory. Do not block CSS or JavaScript that search systems need to render public pages. Do not put secrets, tokens, or private paths in a public robots file.

These public directives guide cooperating crawlers. Disallow is not authentication or access control.

View full-size image

Failure Modes to Recognise

These mistakes are common enough to deserve special attention:

• blocking /assets/, /css/, or /js/ so rendered pages cannot be understood
• adding noindex to a page and then blocking that same page from being fetched
• listing sensitive staging paths in a public robots file
• using Disallow to handle deleted content that should return 404 or 410
• applying broad rules such as Disallow: / during launch and forgetting to remove them
• assuming all crawlers obey robots rules equally

When you audit a rule, write the intended outcome before judging the mechanism. A rule that is wrong for privacy may still be acceptable for crawl hygiene on a large public site. Context matters.