Create a robots.txt file to control search engine crawlers and improve your website's SEO.
How to Use This Robots.txt Generator
- Add your sitemap URL (optional but recommended)
- Click "Add New Rule" to create crawler directives
- For each rule, select the user-agent and specify paths
- Use "Allow" to permit crawling and "Disallow" to block
- Click "Generate robots.txt" to create your file
- Download or copy the generated file
- Upload to your website's root directory (e.g., www.example.com/robots.txt)
Common Robots.txt Rules
- Allow all crawlers: User-agent: * | Disallow: (leave empty)
- Block all crawlers: User-agent: * | Disallow: /
- Block specific folder: User-agent: * | Disallow: /private/
- Allow one crawler only: User-agent: Googlebot | Disallow: (empty) + User-agent: * | Disallow: /
- Block specific file type: User-agent: * | Disallow: /*.pdf$