Robots.txt Configuration: Best Practices and Common Mistakes
Robots.txt controls which parts of your site search engine crawlers can access. A single misconfigured line can deindex your entire site or waste crawl budget on irrelevant pages.
Key Takeaways
- Crawlers request `/robots.txt` before crawling.
- Sitemap: https://example.com/sitemap.xml
- `Disallow: /static/` prevents Google from rendering your pages, leading to incorrect indexing.
- Use Google Search Console's robots.txt tester to verify that important pages are allowed and that private pages are properly blocked.
- Always allow access to CSS, JavaScript, and image files that affect page rendering.
Robots.txt Analyzer
Parse and test robots.txt rules against URLs
How Robots.txt Works
Crawlers request /robots.txt before crawling. The file contains directives that specify which URL paths each crawler (user-agent) may or may not access. It is advisory, not enforced — well-behaved crawlers respect it, malicious ones ignore it.
Basic Syntax
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/internal/
Sitemap: https://example.com/sitemap.xml
Key Directives
| Directive | Purpose |
|---|---|
| User-agent | Which crawler the rules apply to |
| Disallow | Paths the crawler should NOT access |
| Allow | Override a broader Disallow rule |
| Sitemap | Location of your XML sitemap |
| Crawl-delay | Request delay between crawls (not Google) |
Common Mistakes
Blocking CSS and JS
Disallow: /static/ prevents Google from rendering your pages, leading to incorrect indexing. Always allow access to CSS, JavaScript, and image files that affect page rendering.
Disallow vs Noindex
Disallow prevents crawling, not indexing. If other sites link to a disallowed URL, Google may still index it (without visiting it). To prevent indexing, use the noindex meta tag or X-Robots-Tag header instead.
Trailing Slashes
Disallow: /blog blocks /blog, /blog/, and /blog/post-1. Use Disallow: /blog/ (with trailing slash) to block only URLs under the directory while allowing /blog itself.
Wildcard Patterns
Google supports * (match any) and $ (end of URL) wildcards:
Disallow: /*.json$blocks all .json URLsDisallow: /search?*sort=blocks sorted search results
Testing
Use Google Search Console's robots.txt tester to verify that important pages are allowed and that private pages are properly blocked. Check the file after every deployment.