Generate Robots.txt
Generated Output
Copy this into your robots.txt file.
How to Use Robots.txt
A robots.txt file tells search engine crawlers which areas of your website they can access. Place the generated file in your website root directory, for example: https://example.com/robots.txt.
- User-agent targets all crawlers or a specific crawler.
- Disallow blocks crawlers from selected paths.
- Allow permits specific paths.
- Sitemap helps crawlers discover your sitemap.