Advertisement (728x90)
Rules Builder
robots.txt
# Configuration will render here...

Understanding the Robots.txt File

A robots.txt file lives at the root of your website and instructs web robots (typically search engine crawlers) on which pages or files they can or cannot request from your site.

This is crucial for managing crawler traffic, preventing the indexing of sensitive or duplicate pages (like admin dashboards, API routes, or internal search results), and keeping your server load optimized.

Important Security Note: robots.txt is not a security mechanism. Malicious bots can and will ignore these rules. Never use Disallow to hide sensitive data or credentials. Use proper server-side authentication instead.

Core Directives

  • User-agent:
    Targets a specific bot (e.g., 'Googlebot'). Using '*' targets all bots.
  • Disallow:
    Tells the targeted bot not to access a specific URL or directory.
  • Allow:
    Overrides a Disallow directive to permit access to a specific sub-folder or file.
  • Sitemap:
    Points crawlers to your XML sitemap, helping them discover your pages efficiently.