robots.txt (security)
The robots.txt file tells search engines which pages they can visit. This check ensures it doesn't accidentally block the entire site and do…
Analyse my site for freeUnderstanding "robots.txt (security)"
The /robots.txt file is a public text file that tells search engine crawlers which parts of the site they may explore. While primarily an SEO tool, it has two direct security implications: a misconfiguration can block indexing of the entire site, and explicitly listing sensitive paths (/admin, /backup…) effectively hands a map to any attacker who bothers to read it — robots.txt is public by nature, no authentication is required to view it.
This check focuses on these two risks: accidental total blocking and disclosure of sensitive paths, independently of the dedicated SEO check that verifies consistency with the sitemap.
Disallow: /
↳ blocks indexing of the whole site
Disallow: /wp-admin/
Sitemap: https://yoursite.com/sitemap.xml
How TheSiteFuse checks "robots.txt (security)"
TheSiteFuse downloads https://yourdomain/robots.txt. If the file is absent or unreachable, a mild warning is returned (it is not mandatory). The content is then analysed: if both User-agent: * and Disallow: / lines are present, the file blocks the entire site — the check fails with the most severe status (error). Disallow lines mentioning common sensitive paths (/wp-admin, /admin, /login, /checkout, /panier) are also flagged for information.
Why "robots.txt (security)" matters
- Total deindexing — a
Disallow: /forgotten after a staging deployment is one of the most common causes of a site suddenly vanishing from Google results. This mistake, often made during a migration, can go unnoticed for weeks. - Mapping for attackers — listing
/admin,/wp-adminor/backupinDisallowrules does not hide them: it simply tells Google not to index them. Anyone — including an attacker — can read this public file and use it as a starting point for a targeted scan. - False sense of security — robots.txt is absolutely not an access-control mechanism. Believing a listed path is "protected" is a dangerous mistake.
Fix "robots.txt (security)" step by step
Step 1 — Check after every deployment
Add an automated test that fails if Disallow: / appears in production robots.txt. This is the costliest and easiest-to-prevent mistake.
Step 2 — Don't list sensitive paths
To prevent indexing of a sensitive area without publicly mapping it, use instead:
- The
<meta name="robots" content="noindex">tag directly on the pages concerned (requires the page to be reachable by crawlers, but doesn't appear in plain text in a file anyone can view in one click). - Real access protection (authentication, IP restriction) — see the "Exposed admin pages" check.
Clean robots.txt example
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://yoursite.com/sitemap.xml
Reference resource
To deepen your understanding of the technical concepts behind this check, see the dedicated Wikipedia article.
Wikipedia — robots.txt (security)Does your site pass this check?
Run the free full audit (120 checks) and instantly discover what needs fixing.