Skip to content
zspace tools

SEO & Website Tools

Robots.txt Generator

Create a robots.txt file without memorising the syntax. Start from a preset, add user-agent groups and Allow or Disallow rules, declare your sitemap, then test whether a URL is blocked before you upload.

Runs in your browser

Features

What you get

  • Presets

    Allow all, basic website, WordPress and e-commerce examples, custom — and Block all behind a clear confirmation.

  • Visual rule builder

    Add, edit, reorder and remove Allow and Disallow rules for each user-agent group.

  • Multiple crawlers

    Separate groups for *, Googlebot, Bingbot, Googlebot-Image or any custom user agent.

  • Sitemap & extras

    Sitemap declarations, comments and an optional Crawl-delay with a compatibility note.

  • Safety warnings

    Flags site-wide blocks, blocked CSS/JS, relative sitemap URLs and paths that don't start with /.

  • URL tester

    Enter a crawler and a URL to see Allowed or Blocked, with the exact rule that decided it.

How to use

How to use the Robots.txt Generator

Groups start with one or more User-agent lines followed by Allow and Disallow rules; Sitemap lines are listed at the end with full URLs. The tester follows RFC 9309, the Robots Exclusion Protocol: it picks the group naming the crawler (or the * group if none does), finds every rule whose path matches the start of the URL path, and applies the most specific (longest) one — Allow wins a tie. * matches any characters and $ anchors the end. Google's specialised crawlers such as Googlebot-Image fall back to Googlebot's group, as Google documents. It's a simplified model; individual crawlers can differ.

  1. 01

    Pick a starting point

    Enter your site and choose a preset, or start from scratch.

  2. 02

    Adjust the rules

    Add user-agent groups, Allow/Disallow paths and your sitemap URL. Check any warnings.

  3. 03

    Test and upload

    Test important URLs, then download robots.txt and upload it to your site's root.

Examples

Worked examples

  • Most specific rule wins

    With Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php, the AJAX file is allowed and everything else in /wp-admin/ is blocked.

  • End anchor

    Disallow: /*.php$ blocks /index.php but not /index.php?page=2, because $ requires the URL to end there.

  • Crawler-specific group

    A Googlebot group overrides the * group for Googlebot — its rules are not combined with the * rules.

Technical specifications

Directives
User-agent, Allow, Disallow, Sitemap, Crawl-delay
Matching
RFC 9309: longest match, Allow wins ties, * and $
Presets
6 (incl. WordPress and e-commerce examples)
Groups
Unlimited, any user agent
Output
Copy or download robots.txt
Location
https://yourdomain/robots.txt

Privacy

Rules are built and tested in your browser. Nothing is sent or stored.

Read the full privacy policy.

FAQ

Robots.txt Generator questions

A robots.txt file sits at the root of a website and tells compliant crawlers which paths they may fetch. ZSpace Tools' Robots.txt Generator writes the file from a visual rule builder, warns about risky rules such as blocking the whole site or your CSS, and includes a tester that estimates whether a crawler may fetch a given URL.

Groups start with one or more User-agent lines followed by Allow and Disallow rules; Sitemap lines are listed at the end with full URLs. The tester follows RFC 9309, the Robots Exclusion Protocol: it picks the group naming the crawler (or the * group if none does), finds every rule whose path matches the start of the URL path, and applies the most specific (longest) one — Allow wins a tie. * matches any characters and $ anchors the end. Google's specialised crawlers such as Googlebot-Image fall back to Googlebot's group, as Google documents. It's a simplified model; individual crawlers can differ.

No. It controls crawling, not indexing. A blocked URL can still appear in results — usually without a description — if other pages link to it. To keep a page out of results, allow crawling and use a noindex tag, or remove or password-protect it.

Not on the same URL. If robots.txt blocks a page, crawlers can't fetch it, so they never see its noindex tag.

No. robots.txt is public and only a request — anyone can read it, and badly behaved bots ignore it. Protect private content with authentication.

At the root of each host: https://example.com/robots.txt. A subdomain like shop.example.com needs its own file. It must be plain text named exactly robots.txt.

Usually not. Search engines render pages much like a browser, and blocking CSS or JS files can stop them understanding your layout and content.

No. Google ignores Crawl-delay. Bing and some other crawlers honour it. Directive support varies by crawler.