Robots.txt vs noindex: what each directive actually does

Robots.txt primarily controls crawling

A robots.txt rule can tell compliant crawlers not to request particular paths. It is useful for crawl management but is not the same thing as an indexing directive.

Noindex controls indexing

A noindex directive can be sent in HTML meta robots or an X-Robots-Tag HTTP header. For a crawler to reliably process that directive, it normally needs to be able to retrieve the response containing it.

Why combining them can be confusing

If crawling is blocked, the crawler may not retrieve the page and therefore may not see a page-level noindex instruction. That is why diagnosis should check both controls independently rather than assuming “blocked in robots.txt” and “noindex” mean the same thing.

Canonical is different again

A canonical does not prohibit indexing in the way noindex does. It is a signal about which URL should represent similar content.