Add rules
Enter allow and disallow paths.
Code & Data Tool
robots.txt Generator
Generate a basic robots.txt file.
Loading workspace...
About this tool
What is robots.txt Generator?
robots.txt Generator creates a basic robots.txt file that tells crawlers which parts of a site they may visit.
Useful when you need to
- Set up a new site
- Block a private folder from crawlers
- Learn robots.txt structure
How to use
How to use robots.txt Generator
01 02 Generate
Run the tool.
03 Review
Check each rule.
04 Upload
Place the file at the root of your site.
Tips
Get better results
- robots.txt is a request, not security.
- A wrong disallow rule can hide your whole site from search.
FAQ
Frequently asked questions
Does it protect private pages?
No. Use authentication for that.
Is it private?
Yes.
Where does the file go?
At the root, such as example.com/robots.txt.
More practical guides · File handling and privacy · Report an incorrect instruction