Robots.txt Generator: How to Create a Safe File for Your WordPress Blog
Robots.txt is a small plain-text file at the root of your website — yoursite.com/robots.txt — that tells search engine crawlers which parts of the site they may visit and which they should leave alone. A couple of lines in it can open your whole blog to Google — or quietly shut the door.
One point needs saying straight away: robots.txt controls crawling, not indexing. Blocking a page here stops Google visiting it. It does not keep the page out of Google’s index, and it is not a way to remove anything.
If writing the file by hand sounds like an excellent way to ruin a Tuesday, it needn’t be. Our free robots.txt generator lets you choose sensible rules and produces a clean file for you, without you memorising the syntax or worrying that one stray character has blocked your entire site.
Crawling Is Not Indexing: The Distinction That Matters
Crawling is the visit: Googlebot follows links, requests pages and reads what it finds. Indexing is what happens afterwards, when Google processes and stores it for search results. Robots.txt governs only the first step. It can say “don’t visit this”. It cannot say “don’t list this”.
That has an odd consequence. If you disallow a page but other websites link to it, Google can still add it to the index — as a bare link, usually with no title or description, because it was never allowed in to read the page. The page you thought you’d hidden can still surface in a search, looking half-finished.
If you want a page out of the index, use noindex on the page itself, password-protect it, or delete it. Don’t combine the half-measures: block crawling and add noindex, and Google may never see the noindex.
What Each Directive in Robots.txt Actually Does
The file is a short list of instructions, grouped by bot — five or six items cover nearly everything a blogger needs.
User-agent and Disallow
User-agent names the bot the rules apply to. User-agent: Googlebot targets Google specifically; User-agent: * uses the asterisk wildcard to mean every bot.
Disallow tells that bot not to crawl a path: Disallow: /wp-admin/ blocks the WordPress admin area. Paths start with a slash. And the line that should make any owner sit up straight is Disallow: / — that single slash means the entire site. Correct on a private staging site; on a live one, it’s a silent catastrophe.
Allow, Wildcards and the $ Anchor
Allow is the exception-maker: it lets a bot crawl one thing inside an otherwise blocked area. Where rules conflict, the more specific one generally wins — which is what makes the example below work.
The * wildcard stands for any sequence of characters, so Disallow: /*?s= can block internal search URLs with query strings. The $ anchor marks the end of a URL, so Disallow: /*.pdf$ targets files ending in .pdf, not URLs containing those letters mid-address.
The Sitemap Line and Comments
The Sitemap line points bots to your sitemap: Sitemap: https://yoursite.com/sitemap_index.xml. Anything after a # is a comment — ignored by bots, meant for humans. Use them: future you will thank past you for a note instead of a mystery.
A Real Robots.txt Example, Explained Line by Line
Here is a small, sensible file for a typical WordPress blog — nothing fancy, and that’s the point:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap_index.xmlLine by line:
User-agent: *— these rules apply to every crawler, not just Google.Disallow: /wp-admin/— keep bots out of the admin area; WordPress includes this in its default file anyway.Allow: /wp-admin/admin-ajax.php— the exception. Themes and plugins use this file to make parts of the front end work, so allowing it lets Google render your pages properly.- The blank line ends the group — a small thing that keeps the file tidy.
Sitemap: https://yoursite.com/sitemap_index.xml— here is the map of the site, please use it. Swap in your own domain and sitemap address.
That’s it. No rule blocks a post, page, category or image — because being crawled is what you want.
Sensible Robots.txt Rules for a WordPress Blog
What is worth disallowing
The admin area, covered above. Your login page, /wp-login.php, is another reasonable block: no value in search results, and bots have no business there — though, as the next section explains, that’s housekeeping, not protection. Internal search results pages are worth blocking too: they’re usually thin, endless in number, and a poor landing experience.
Be conservative: if it wouldn’t really be a problem for Google to crawl it, leave it alone.
What you must never disallow
Never block your CSS, JavaScript or image files. Google renders pages much as a browser does, and it needs those assets to see your pages properly. Block the stylesheet and Google may judge a tidy, mobile-friendly post as a broken blank page — and rank it accordingly. This turns up often, usually as a leftover line blocking /wp-content/ or /wp-includes/, which hold your theme files, plugin assets and uploaded images.
Never block posts, pages or categories you want found. And never copy a big website’s impressive-looking robots.txt: their rules fit their site, not yours. Pasted into yours, it can block your uploads, your shop, or everything.
Robots.txt Is Not a Lock: The Security Myth
Treat robots.txt as a polite notice on a gate, not the gate itself. Respectable crawlers — Googlebot, Bingbot and the rest — obey it. Bad bots ignore it, and some read it as a shopping list, because the file is public.
So never use it to “hide” sensitive paths. Disallow: /private-client-files/ doesn’t protect that folder; it advertises where it lives. If something shouldn’t be public, use a password, proper permissions, or take it off the public site.
Crawl Budget, in Plain Terms (and Without the Hype)
Crawl budget is the idea that Google will only visit so many pages in a given period, so visits shouldn’t be wasted on junk. It’s real — for very large sites. News publishers with millions of pages, or huge shops whose filters generate endless URL combinations, genuinely need to steer crawlers away from rubbish.
For a blog with a few hundred posts, it barely matters. Google can crawl your whole site without breaking a sweat, and blocking a search page won’t lift your rankings.
How to Use the Robots.txt Generator
Open the robots.txt generator and build the file by choosing rules rather than typing it from memory.
Work through it in this order:
- Start with the default: allow everything. For most blogs the starting point is an open site.
- Add the standard WordPress blocks. Disallow /wp-admin/, keep the allow exception for admin-ajax.php, and add your login page if you wish.
- Add extras sparingly. Internal search results, junk paths your theme generates — one rule at a time.
- Add your sitemap address, using the exact URL your SEO plugin gives you.
- Generate and copy the file. Read it once first; that read is your cheapest safety check.
- Put it in place. Upload it as robots.txt to your site’s root, or paste it into your SEO plugin’s robots.txt editor. Then check yoursite.com/robots.txt.
That last check is not optional: the file that counts is the one at that address, not the one in your editor.
WordPress, the Virtual File, and Checking What You Already Have
A WordPress quirk worth knowing: if no physical robots.txt file exists, WordPress generates a virtual one on the fly, typically allowing everything except /wp-admin/. A real file uploaded to your root replaces the virtual one, and plugins such as Rank Math and Yoast offer editors too. With several possible authors, always judge by what’s served at the address itself.
Checking takes five minutes. Fetch yoursite.com/robots.txt and read it, looking for a whole-site disallow or blocked assets. Then check Google Search Console’s robots.txt report — under Settings, it shows the file Google fetched and any problems found — and use Search Console’s robots.txt tester to test specific URLs against your rules before changing them. Robots.txt sits alongside the other unglamorous foundations in our WordPress technical SEO checklist.
Robots.txt and the robots meta tag on a page are separate instructions. If one page is missing while the rest are fine, run that page through a meta checker to look for a stray noindex — that culprit lives on the page itself, not in your robots.txt file.
If you’ve just removed an accidental block, don’t expect Google to notice the same afternoon; it recrawls on its own schedule. You can hurry things along by submitting the URL through our Google indexing tool, which asks Google to crawl it — a request, not a guarantee.
Common Robots.txt Mistakes That Quietly Cause Damage
- Blocking the whole site with
Disallow: /. The classic — usually a staging file copied to production, or a setting switched during a redesign. - Blocking CSS and JavaScript. Google can’t render the page, judges what it half-sees, and your rankings pay for it.
- Trying to remove a page with Disallow. It blocks the visit, not the listing. Use noindex, or delete the page.
- Hiding sensitive folders in the file. You’ve published a map to them. Use passwords and permissions instead.
- Copying a big site’s file wholesale. Their rules, their problems, your breakage.
- Never testing after editing. Every change deserves two checks: read the live file, and confirm Search Console has no complaints.
Frequently Asked Questions
If I block a page in robots.txt, will it disappear from Google?
Not reliably. Blocking stops Google crawling the page, but if other sites link to it, Google can still index the URL as a bare result with no description. To remove a page, use noindex or delete it.
What happens if my robots.txt says Disallow: / ?
Every well-behaved crawler will refuse to crawl your entire site. Indexed pages won’t vanish overnight, but nothing new will be picked up, and existing pages will gradually drop. If you find this line on a live site, remove it promptly.
Does WordPress create a robots.txt file automatically?
Sort of. With no physical file present, WordPress serves a virtual one allowing everything except /wp-admin/. Plugins can edit it, and uploading a real file replaces the virtual one.
Can robots.txt protect private pages or folders?
No. It’s a public file that polite bots obey and impolite ones ignore. Use passwords and proper permissions for anything that needs protecting.
How do I know if my robots.txt is blocking something important?
Read the file at yoursite.com/robots.txt, check Search Console’s robots.txt report for problems, and test important URLs against your rules. If one page is missing, also check it for a noindex tag.
Final Thoughts
Robots.txt rewards a light touch. For most blogs, the ideal file is short, mostly permissive and slightly boring: admin closed, one sensible exception, sitemap attached, nothing creative done to the content. The danger was never too few rules — it’s one bad line nobody checked, quietly undoing months of work.
So make yours with the robots.txt generator, read it once at the live address, and check it again after any redesign, migration or staging-site adventure. Two minutes of checking beats learning about Disallow: / from your traffic graph.
