Skip to content
Get your free SEO report→
GeneratorBeginner

Robots.txt Generator

Generate a valid robots.txt file in seconds — pick your crawler rules, add your paths, and copy the result.

Share:

Build your robots.txt

Sitemap directives

A

100/100 — looks good

Syntax check passed — safe to ship.

Preview 
User-agent: Googlebot
Allow: /
Disallow: /admin/
Disallow: /tmp/

User-agent: *
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Test a path against your rules

Testing against: Googlebot group

Blocked for Googlebot

Blocked by "/admin/".

What a robots.txt file actually does

A robots.txt file tells crawlers which pages they are allowed to visit and which they must skip. It lives at the root of your domain — https://yoursite.com/robots.txt — and is the first thing most crawlers fetch before they touch any other page. A well-structured file keeps your staging environment out of the index, protects your admin routes from being crawled, and points crawlers to your sitemap in one line.

Directive order matters

The RFC 9309 standard requires crawlers to read each User-agent group in order, merging rules for the same bot, then applying the most specific path pattern. A rule at the top of the file can be overridden by a more specific rule lower down — order affects the outcome. This generator emits directives in the correct RFC order so crawlers interpret them as you intend.

What to block on every site

  • Admin and login pages — /wp-admin/, /admin/,/login/. These waste crawl budget and should never appear in search results.
  • Internal search and filter results — /search/,/?*filter=. These generate infinite unique URLs that dilute your crawl budget.
  • Staging and dev environments — block the entire staging subdomain so Google never indexes a duplicate version of your site.
  • Private or temporary pages — /tmp/,/pending-review/. Pages that exist briefly before going live.

Wildcard patterns

The * character matches any sequence of characters. The $ anchor marks the end of the URL. So Disallow: /*.pdf$ blocks every PDF on your site without affecting other file types. This generator adds $ automatically when you paste an extension pattern so you do not accidentally block all URLs that contain that string.

What robots.txt cannot do

A robots.txt is a request, not a security barrier. Sophisticated crawlers — and all malicious bots — ignore it entirely. Do not use robots.txt to protect sensitive content; use authentication, a meta noindex tag, or an X-Robots-Tag header instead. For truly private pages, keep them behind a login or firewall.