What is robots.txt?

The robots.txt file is a text file placed at the root of your site (https://your-site.com/robots.txt) that communicates to search engine robots — crawlers (such as Googlebot, Bingbot, DuckDuckBot) — the rules they should follow when exploring your site. It is a courtesy protocol: well-behaved robots respect it, but it has no legal value and does not prevent a malicious robot from ignoring these rules.

Structure of a robots.txt file

# Rules for all robots
User-agent: *
Disallow: /admin/
Disallow: /account/
Disallow: /api/
Allow: /

# Rules specific to Googlebot
User-agent: Googlebot
Allow: /blog/

# Point to sitemap
Sitemap: https://your-site.com/sitemap.xml
  • User-agent: *: applies to all robots
  • Disallow: /path/: forbids crawling this path (trailing slash includes all sub-paths)
  • Allow: /path/: explicitly allows (useful to refine a broad Disallow rule)
  • Sitemap:: indicates the location of sitemap.xml — very useful for Googlebot

What robots.txt CANNOT do

Blocking a page in robots.txt does not prevent it from being indexed if other sites link to it. Google can index a page without ever crawling it. To definitively exclude a page from the index, use the meta tag:

<meta name="robots" content="noindex, nofollow">

The fatal mistake: blocking CSS and JavaScript

Google needs to see your CSS and JS to understand your content and evaluate user experience. Blocking these files in robots.txt prevents Googlebot from correctly "rendering" the page, which can degrade your SEO rankings. Avoid Disallow: /static/ or Disallow: /assets/.

Crawl budget and optimisation

The crawl budget is the number of pages Googlebot explores on your site in a given time. Blocking pages with no SEO value (pagination, filter pages, admin pages) allows Googlebot to concentrate on your main content and speeds up the indexation of new pages.

Check your robots.txt

TheSiteFuse audits your robots.txt file at every scan: file presence, Sitemap directive, consistency with your sitemap.xml, and absence of dangerous blocks on CSS/JS. Run a free audit for a complete diagnosis.