Robots.txt Generator
Build a valid, SEO-friendly robots.txt file in seconds. Control crawler access, protect sensitive directories, and point bots to your sitemap.
Configure Your robots.txt
Quick Presets
Global Settings
Sets a pause between requests. Most sites don't need this.
Allow / Disallow Rules
For User-agent: * (all crawlers). Use /path/ format.
Block Specific Bots
Generated robots.txt
# Click "Generate robots.txt" to build your file
Validation Checks
Generate your file firstValidation results will appear here after you generate your robots.txt.
How to upload your robots.txt
- Download the file using the button above
- Upload it to the root directory of your website (same folder as your homepage)
- Verify it's accessible at
https://yoursite.com/robots.txt - Submit your sitemap URL in Google Search Console
What it does
Controls crawler access
robots.txt is the first file search engine bots fetch. It tells them which pages they can and cannot crawl.
What it does not do
It's not a security tool
Disallowed pages can still be indexed if other sites link to them. Use authentication or noindex for true blocking.
File location
Must be at the root
The file must be at
yoursite.com/robots.txt — not in a subfolder or subdomain.Want a fully optimised technical SEO setup?
We handle robots.txt, sitemap.xml, schema markup, Core Web Vitals — everything, done right from day one.