A robots.txt file is a plain-text convention rather than a standard with an enforcement body. It lives at one place, the root of the host — https://example.com/robots.txt — and is read before any other path on that host is requested. A copy in a subdirectory is not consulted, and one served through a redirect to another host is usually treated as absent.
The distinction that matters: robots.txt governs fetching, not indexing. A crawler told not to fetch a path never sees the page, but if other pages link to it the engine can still list the address.
A group is one or more User-agent lines followed by rules. A blank line ends a group, so an empty line in the middle of your rules orphans everything below it.
| Directive | Takes | Effect |
|---|---|---|
User-agent | A crawler token such as Googlebot, or * | Starts a group. * is the fallback for crawlers named in no other group. |
Disallow | A path pattern; empty means nothing | Blocks matching paths. An empty value is a no-op, which is how one agent is let in under a looser blanket rule. |
Allow | A path pattern | Re-permits a subpath beneath a broader Disallow. |
Sitemap | A full absolute URL | Belongs to no group. Lists a sitemap for discovery; wildcards are not permitted here. |
The crawler picks one group: the one whose User-agent token is the longest match for its own name, falling back to *. Within it, Google documents choosing the most specific rule — the longest matching path — and, where an Allow and a Disallow are equally specific, the less restrictive of the two. Many other crawlers follow the older reading, in which the first rule matching the path wins. Write rules whose outcome does not depend on which reading is used.
Matching is a case-sensitive prefix comparison on the request path. Disallow: /private blocks /private/, /private/notes and /private-archive. * stands for any run of characters and a trailing $ anchors to the end of the URL; both are widely supported but were never part of the original draft, so they are not access control.
Given * with an allow of / and a disallow of /admin, the output is those three directives followed by Sitemap: https://example.com/sitemap.xml. An empty path box is written through as /, so a blank rule produces the broadest possible rule.
Disallow: / from staging. The classic failure: the site ships, the file goes with it, and every crawler is turned away at the door. Existing listings linger, then drop out with nothing in Search Console pointing at the cause./assets/ or /js/ stops the crawler fetching stylesheet and scripts, so the renderer sees a layout that is not the one users see.noindex from the page — noindex written inside robots.txt is not honoured by the major crawlers.Disallow prevent indexing?It prevents fetching. A URL can be listed without its content ever being read. For exclusion from results, allow the crawl and serve noindex from the page.
Yes. Disallow: /blog/ blocks the directory, while Disallow: /blog also blocks /blogger and /blog-archive. When you mean a directory, write the slash.
It is redundant once the engine knows the sitemap, but cheap, and it covers a new crawler arriving with no prior knowledge of the site.
No. The file is read per host, so www.example.com and shop.example.com each need their own, and a rule written on one does not reach the other.