Your Clicks, Our Hype
CLICK HYPE TOOLS
Robots.txt Generator
Build a valid robots.txt file visually — no code to learn. Pick a preset, tweak the rules, add your sitemap, then copy or download the finished file.
1. Start from a preset
Each preset loads an editable starting point below. Nothing is final until you copy or download it.
2. Edit the rules
Each group applies to one crawler (user-agent). Paths must start with / — the generator fixes them automatically if you forget.
Sitemaps
Full sitemap URLs, one per line. These are added as Sitemap: lines at the end of the file.
3. Your robots.txt
Updates live as you edit.
robots.txt
Upload the downloaded file to the root of your domain so it lives at
yourdomain.com/robots.txt. On WordPress, Rank Math and Yoast can also generate and edit this file for you.What is robots.txt — and what is it not?
robots.txt is a small text file that sits at the root of your website and politely tells search-engine crawlers and other bots which parts of the site they may — and may not — crawl. It is read before a well-behaved crawler visits your pages, so it is the standard way to keep admin areas, internal search results and duplicate or low-value paths out of crawl queues and to save your crawl budget for the pages that matter.
What robots.txt is NOT
- It is a request, not security. Polite crawlers honour it; malicious bots ignore it completely. Never rely on robots.txt to hide sensitive files — anything listed in it can still be visited directly, and the file itself is public, so listing a "secret" folder actually advertises it.
- It does not remove pages from Google. Blocking a page that is already indexed usually keeps it in the index (shown without a description), because Google can no longer crawl it to see a "noindex" tag. To remove a page, use a password, delete it, or add a genuine
noindex— and leave it crawlable. - It is not a ranking switch. It controls crawling, not indexing or rankings directly, though sensible rules help crawlers spend their time on your important pages.
Where the file lives
There is exactly one robots.txt per domain (including each subdomain), at the root: https://yourdomain.com/robots.txt. It must be a plain text file, publicly reachable, and return a normal 200 status. Rules are case-sensitive for paths, directives apply per user-agent group, and the most specific matching rule wins — which is why the WordPress preset allows /wp-admin/admin-ajax.php even though /wp-admin/ is disallowed.
Before you go live
Read Google's own introduction to robots.txt, then test your file in Google Search Console (Settings → robots.txt tester / the URL inspection tool) to confirm Googlebot can still reach your important pages. Be especially careful with any Disallow: / rule — a single misplaced line can ask Google to stop crawling your entire site.