SEO & Website Tools
Robots.txt Generator
Create a robots.txt file without memorising the syntax. Start from a preset, add user-agent groups and Allow or Disallow rules, declare your sitemap, then test whether a URL is blocked before you upload.
Runs in your browser
Features
What you get
Presets
Allow all, basic website, WordPress and e-commerce examples, custom — and Block all behind a clear confirmation.
Visual rule builder
Add, edit, reorder and remove Allow and Disallow rules for each user-agent group.
Multiple crawlers
Separate groups for *, Googlebot, Bingbot, Googlebot-Image or any custom user agent.
Sitemap & extras
Sitemap declarations, comments and an optional Crawl-delay with a compatibility note.
Safety warnings
Flags site-wide blocks, blocked CSS/JS, relative sitemap URLs and paths that don't start with /.
URL tester
Enter a crawler and a URL to see Allowed or Blocked, with the exact rule that decided it.
How to use
How to use the Robots.txt Generator
Groups start with one or more User-agent lines followed by Allow and Disallow rules; Sitemap lines are listed at the end with full URLs. The tester follows RFC 9309, the Robots Exclusion Protocol: it picks the group naming the crawler (or the * group if none does), finds every rule whose path matches the start of the URL path, and applies the most specific (longest) one — Allow wins a tie. * matches any characters and $ anchors the end. Google's specialised crawlers such as Googlebot-Image fall back to Googlebot's group, as Google documents. It's a simplified model; individual crawlers can differ.
- 01
Pick a starting point
Enter your site and choose a preset, or start from scratch.
- 02
Adjust the rules
Add user-agent groups, Allow/Disallow paths and your sitemap URL. Check any warnings.
- 03
Test and upload
Test important URLs, then download robots.txt and upload it to your site's root.
Examples
Worked examples
Most specific rule wins
With Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php, the AJAX file is allowed and everything else in /wp-admin/ is blocked.
End anchor
Disallow: /*.php$ blocks /index.php but not /index.php?page=2, because $ requires the URL to end there.
Crawler-specific group
A Googlebot group overrides the * group for Googlebot — its rules are not combined with the * rules.
Technical specifications
- Directives
- User-agent, Allow, Disallow, Sitemap, Crawl-delay
- Matching
- RFC 9309: longest match, Allow wins ties, * and $
- Presets
- 6 (incl. WordPress and e-commerce examples)
- Groups
- Unlimited, any user agent
- Output
- Copy or download robots.txt
- Location
- https://yourdomain/robots.txt
Privacy
Rules are built and tested in your browser. Nothing is sent or stored.
Read the full privacy policy.
FAQ
Robots.txt Generator questions
A robots.txt file sits at the root of a website and tells compliant crawlers which paths they may fetch. ZSpace Tools' Robots.txt Generator writes the file from a visual rule builder, warns about risky rules such as blocking the whole site or your CSS, and includes a tester that estimates whether a crawler may fetch a given URL.
Groups start with one or more User-agent lines followed by Allow and Disallow rules; Sitemap lines are listed at the end with full URLs. The tester follows RFC 9309, the Robots Exclusion Protocol: it picks the group naming the crawler (or the * group if none does), finds every rule whose path matches the start of the URL path, and applies the most specific (longest) one — Allow wins a tie. * matches any characters and $ anchors the end. Google's specialised crawlers such as Googlebot-Image fall back to Googlebot's group, as Google documents. It's a simplified model; individual crawlers can differ.
No. It controls crawling, not indexing. A blocked URL can still appear in results — usually without a description — if other pages link to it. To keep a page out of results, allow crawling and use a noindex tag, or remove or password-protect it.
Not on the same URL. If robots.txt blocks a page, crawlers can't fetch it, so they never see its noindex tag.
No. robots.txt is public and only a request — anyone can read it, and badly behaved bots ignore it. Protect private content with authentication.
At the root of each host: https://example.com/robots.txt. A subdomain like shop.example.com needs its own file. It must be plain text named exactly robots.txt.
Usually not. Search engines render pages much like a browser, and blocking CSS or JS files can stop them understanding your layout and content.
No. Google ignores Crawl-delay. Bing and some other crawlers honour it. Directive support varies by crawler.
Related tools
XML Sitemap Generator
Build a valid sitemap.xml from a URL list — with dates, splitting and a sitemap index.
Meta Tag Generator
Create title, description, canonical, robots, Open Graph and X tags with live previews.
Open Graph Preview
Preview link cards for Facebook, LinkedIn, WhatsApp and X, and generate the tags.