robots.txt Configuration Guide: Syntax, Rules, and Troubleshooting

robots.txt is the first "agreement" between a website and search engine crawlers. It tells crawlers which paths can be crawled and which should not be accessed. Proper configuration guides crawlers to efficiently crawl important content and conserve crawl budget; incorrect configuration can prevent your entire site from being indexed. This article covers syntax basics through practical configurations to help you write safe, effective robots.txt files.

What is robots.txt#

robots.txt is a plain text file placed in the website root directory, accessible directly via https://example.com/robots.txt. It follows the Robots Exclusion Protocol (RFC 9309) standard and is one of the first files search engine crawlers request when visiting a site.

When Googlebot or other crawlers prepare to crawl a URL, they first check the site's robots.txt to confirm whether the target path is allowed. If robots.txt explicitly prohibits that path, compliant crawlers will skip the URL without making an actual request.

How It Works#

The complete crawler workflow for robots.txt:

1. Request robots.txt: Before crawling any page on the site, crawlers first request /robots.txt.
2. Cache rules: Crawlers cache the robots.txt content (Google typically caches for ~24 hours) and don't re-request for every crawl.
3. Match User-agent: Crawlers find the User-agent section matching their identity and read the Disallow and Allow rules within.
4. Evaluate path: The target URL path is matched against rules sequentially to decide whether to crawl.

Basic Syntax#

robots.txt consists of one or more "rule groups," each starting with a User-agent line, followed by Disallow and/or Allow directives. Sitemap directives can be placed anywhere in the file.

# This is a comment, crawlers ignore it

User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /admin/public/

User-agent: Googlebot
Disallow: /tmp/

Sitemap: https://example.com/sitemap.xml

Directive Explained

Path Matching Rules#

robots.txt path matching is prefix-based: if the rule path is a prefix of the target URL path, it's a match. Google and Bing also support two extended wildcards:

When the same URL matches both Disallow and Allow, Google uses most specific rule wins (longer path wins). If lengths are equal, Allow takes precedence.

User-agent: *
Disallow: /products/
Allow: /products/featured/

# /products/list.html       → Blocked (matches Disallow: /products/)
# /products/featured/new    → Allowed (Allow path is longer, more specific)
# /products/                → Blocked

Common Configuration Scenarios#

Below are the most common robots.txt configuration needs in real projects:

Block backend and admin pages

User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /dashboard/

Block search results and filter pages

Internal search result pages and pages with numerous filter combinations are the biggest waste of crawl budget.

User-agent: *
Disallow: /search
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /*&sort=
Disallow: /*&filter=

Block specific file types

User-agent: *
Disallow: /*.pdf$
Disallow: /*.doc$

Allow all crawling (no restrictions)

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Set different rules for different crawlers

# Google can crawl all content
User-agent: Googlebot
Disallow:

# Other crawlers block private paths
User-agent: *
Disallow: /private/
Disallow: /staging/

robots.txt vs noindex#

This is one of the most commonly confused concepts in technical SEO. While both involve how search engines handle pages, they solve completely different problems.

Decision path is simple:

• Want to save crawl budget and focus crawler resources on important pages → Use robots.txt
• Want page to not appear in search results → Use noindex (and ensure robots.txt doesn't block it)
• Want page to not be accessed by anyone → Use password protection or server access control

Common Crawler User-agents#

Different search engines use different User-agent identifiers. Understanding them helps write targeted rules.

Testing with Google Search Console#

Google Search Console provides a robots.txt testing tool that lets you verify rules work as expected before modifying the file.

1. Open Google Search Console, find "Settings" → "robots.txt" in the left menu (or access via the legacy tool at search.google.com/search-console/robots-testing-tool).
2. The tool displays your current robots.txt content and highlights parsing results.
3. Enter a URL path in the bottom input box and click "Test". The tool tells you whether the path is allowed or blocked, and which rule matched.
4. You can modify rule content directly in the editor for testing, but changes aren't auto-saved—you need to manually update the file on your server.

Beyond Search Console, you can also visit https://your-domain/robots.txt directly in a browser to check if the file is accessible and content is correct.

For more details, see Google official documentation: Create a robots.txt file.

Common Errors#

robots.txt syntax is simple, but errors can have serious consequences. Here are the most common issues:

1. Accidentally blocking entire site

# This blocks all paths for all crawlers!
User-agent: *
Disallow: /

Most common scenario: testing environment uses Disallow: /, forgotten when deploying to production.

2. Blocking CSS and JavaScript resources

If robots.txt blocks CSS or JavaScript files, Google's rendering service cannot properly render the page, potentially failing to understand page content and layout, affecting indexing quality.

# Don't do this—Google needs these resources to render pages
User-agent: *
Disallow: /assets/css/
Disallow: /assets/js/

3. Placing robots.txt in wrong location

robots.txt must be in the domain root directory. https://example.com/blog/robots.txt won't be recognized by any crawler. Each subdomain needs its own robots.txt (https://blog.example.com/robots.txt).

4. Mixing robots.txt and noindex

Explained in detail above—blocking a page with robots.txt while expecting noindex to work is contradictory.

5. Using Crawl-delay expecting to control Google crawl frequency

Google doesn't support the Crawl-delay directive. If you need to control Google's crawl rate, configure it in Google Search Console. Crawl-delay works for Bing and Yandex.

Subdomains and Multi-site#

robots.txt scope is per origin (protocol + host + port). This means:

• https://example.com/robots.txt only controls paths under https://example.com
• https://blog.example.com needs its own /robots.txt
• http://example.com and https://example.com are different origins, each needing its own robots.txt (though you typically redirect HTTP to HTTPS via 301)

If a subdomain has no robots.txt (returns 404), crawlers treat it as having no restrictions, allowing all paths to be crawled.

This Site's robots.txt#

MagicSEO is a pure static content site where all pages should be indexed by search engines, so robots.txt is very simple:

User-agent: *
Disallow:

Sitemap: https://magic-seo.com/sitemap.xml

Configuration Checklist#

• robots.txt placed in domain root, returns 200
• No accidental Disallow: / blocking entire site
• Not blocking CSS, JavaScript, or other rendering resources
• Not using robots.txt block and noindex on same page simultaneously
• Sitemap URLs use complete absolute URLs
• Verify key URLs with GSC robots.txt testing tool
• Each subdomain has its own robots.txt
• Check robots.txt is updated when moving from test to production
• Regularly review to ensure rules still match current site structure

Frequently Asked Questions#

Can robots.txt prevent pages from appearing in search results?

Not reliably. robots.txt prevents crawling, not indexing. If other sites link to a URL blocked by robots.txt, search engines may still display that URL in search results based on the link anchor text (though without a snippet). To prevent indexing, use meta robots noindex or X-Robots-Tag: noindex, and don't block the page with robots.txt.

Where must the robots.txt file be located?

It must be in the website root directory, accessible directly via https://example.com/robots.txt. Placing it in a subdirectory (such as /blog/robots.txt) is invalid. Each subdomain (such as blog.example.com) needs its own robots.txt.

What happens if robots.txt is written incorrectly?

If there are syntax errors, different search engines may handle parsing differently—some ignore the error line, others may ignore the entire file. The most dangerous error is accidentally blocking important pages or the entire site (such as root directory Disallow: /), which prevents search engines from crawling any content. Always verify with Google Search Console's robots.txt testing tool after making changes.

Does Google support the Crawl-delay directive?

Google does not support the Crawl-delay directive in robots.txt and will ignore it. Bing and Yandex do support it. If you need to control Google's crawl rate, adjust the crawl rate setting in Google Search Console.