What Is a robots.txt File?
A robots.txt file is a plain text document placed at the root of a website that instructs web-crawling software (such as Googlebot, Bingbot, and advertising review bots) which URLs or sections of your site they can or cannot request. Governed internationally by the Robots Exclusion Protocol (formalized in RFC 9309), it serves as the foundational gatekeeper for search engine indexing and crawler resource management.
Whenever an automated search engine spider visits a website, its very first request is almost always to /robots.txt. If the file grants permission, the crawler proceeds to fetch your web pages. If the file forbids access to specific directories or URLs, search engines will respect those boundaries and avoid scanning the disallowed paths.
How to Use This Free Robots.txt Generator
Creating an optimized robots.txt file using Rankzio takes four straightforward steps:
- Select a Starting Preset: Click on Default Preset, WordPress Preset, or AdSense Ready to instantly populate standard industry best-practice rules.
- Configure Directories: Add any private folders you want kept away from search engines (such as
/admin/,/cgi-bin/, or user checkout carts) in the Disallowed Directories box. Add any explicit permissions in the Allowed Directories box. - Specify Your XML Sitemap: Enter your live sitemap address (e.g.,
https://rankzio.online/sitemap.xml) so crawlers automatically discover all your canonical URLs. - Copy or Download: Click "Copy robots.txt" or "Download .txt" and upload the file to your root web folder (e.g.,
public_html/robots.txt).
Core Directives Explained: User-agent, Allow, Disallow, Sitemap
A valid robots.txt file is structured around four primary directives:
-
User-agent:Defines which specific search crawler the following rules apply to. Using an asterisk (User-agent: *) applies the rules universally to all automated crawlers. Specific identifiers likeGooglebot,Bingbot, orMediapartners-Googletarget individual bots. -
Disallow:Tells the crawler not to access pages matching the specified path. For example,Disallow: /admin/prevents crawlers from fetching any URL beginning with/admin/. A blank disallow (Disallow:) explicitly allows everything. -
Allow:Used to grant access to a specific subfolder or file that would otherwise be blocked by a parent Disallow rule. For example, if you disallow/wp-admin/, you can addAllow: /wp-admin/admin-ajax.phpso that dynamic AJAX requests can still function. -
Sitemap:Points search engines directly to the location of your XML sitemap. Unlike Allow and Disallow directives, the Sitemap line must always be an absolute URL (e.g.,https://rankzio.online/sitemap.xml).
Practical Robots.txt Examples
Here is an industry-standard, clean configuration suitable for modern content publishers and webmasters:
# Standard Robots.txt for Rankzio
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Allow: /public/
# Explicitly allow Google AdSense review crawler
User-agent: Mediapartners-Google
Disallow:
Sitemap: https://rankzio.online/sitemap.xmlThe Critical Difference Between robots.txt and "noindex"
CRITICAL DISTINCTION: robots.txt is NOT a de-indexing mechanism!
Many website owners mistakenly believe that adding Disallow: /secret-page in robots.txt will remove the page from Google Search results. This is false! A disallow directive prevents Google from crawling the page, but if another site links to that URL, Google can still index the URL without reading its content, resulting in search results reading: "A description for this result is not available because of this site's robots.txt". If you want a page permanently excluded from search results, allow it in robots.txt and use a <meta name="robots" content="noindex"> header tag instead!
Why You Must NEVER Block CSS and JavaScript Files
Historically, webmasters blocked folders containing CSS stylesheets, JavaScript files, and theme images (such as /wp-content/themes/ or /public/assets/). However, modern search engines—particularly Googlebot—render pages like a modern headless Chrome browser.
When Googlebot crawls a web page, it executes JavaScript and renders the CSS stylesheet to assess visual layout, mobile viewport responsiveness, and Core Web Vitals (LCP, CLS). If your robots.txt file blocks your CSS or JS files, Google cannot render the page properly, causing the mobile-friendly evaluation to fail and severely damaging your organic search rankings.
Common Mistakes to Avoid in robots.txt
- Accidental Global Block: Writing
Disallow: /underUser-agent: *blocks search engines from your entire website. Always verify that a single slash isn't left after Disallow unless you genuinely want the site hidden. - Case Sensitivity Errors: Unix and Linux web servers are case-sensitive. Disallowing
/admin/will NOT disallow/Admin/or/ADMIN/. Match your URL casing exactly. - Blocking Mediapartners-Google: Google AdSense uses a dedicated user-agent called
Mediapartners-Googleto review web pages for ad suitability. If your site blocks this bot, your AdSense application will be rejected with "Site Down or Unavailable". - Placing in Subdirectories: A robots.txt file located at
https://yourdomain.com/blog/robots.txtwill be completely ignored by search engines. The file MUST reside at the domain root:https://yourdomain.com/robots.txt. - Relative Sitemap Paths: The Sitemap directive must always include the full protocol and domain (e.g.,
https://yourdomain.com/sitemap.xml), never a relative path like/sitemap.xml.
Technical Specifications (RFC 9309 Compliance)
The official Internet Engineering Task Force (IETF) specification RFC 9309 defines the following technical rules for robots.txt:
- File Size Limit: Google enforces a parsing cap of 512 kibibytes (KiB). Any directives beyond 512 KiB are ignored.
- Character Encoding: Must be encoded in standard UTF-8 format without BOM (Byte Order Mark).
- Wildcards & End of URL: Supports the wildcard asterisk (
*) for matching arbitrary sequences and the dollar sign ($) to mark the exact end of a path pattern (e.g.,Disallow: /*.pdf$). - Precedence: When both Allow and Disallow directives match a specific URL, crawlers choose the directive with the longest character pattern match.
Frequently Asked Questions (FAQ)
Where do I upload the robots.txt file?
You must upload robots.txt to the highest root directory of your website domain, typically located in public_html or htdocs. It must be accessible directly at https://yourdomain.com/robots.txt.
What happens if my website has no robots.txt file?
If a crawler requests /robots.txt and receives a 404 Not Found error, it interprets that as complete permission to crawl every publicly discoverable page on your website. While this is not fatal, having a clean robots.txt allows you to protect private admin areas and reference your XML sitemap.
Does Googlebot obey the Crawl-delay directive?
No. Googlebot ignores the Crawl-delay directive in robots.txt. If Googlebot is crawling your server too aggressively and causing high CPU usage, you can adjust your crawl rate inside Google Search Console settings. Other search engines like Bing and Yandex still respect Crawl-delay.
Can I use robots.txt to keep confidential data secure?
Never. Because robots.txt is a publicly readable file, anyone on the internet can view it in their browser. Listing private URLs or sensitive directory names in robots.txt actually advertises their locations to malicious actors. Use password protection (HTTP Basic Auth) or access control lists to secure confidential data.
How does the AdSense crawler (Mediapartners-Google) interact with robots.txt?
Mediapartners-Google is the specialized crawler used by Google AdSense to audit content for policy compliance and serve targeted ads. If you block general crawlers in your robots.txt, always include a dedicated block allowing User-agent: Mediapartners-Google with Disallow: to ensure AdSense can review your pages.
How long does it take for Google to recognize changes to robots.txt?
Googlebot typically caches your robots.txt file for up to 24 hours. If you need immediate refresh after fixing a critical crawl error, you can use the Google Search Console robots.txt report to request an expedited re-crawl.
Can I disallow specific file types like PDFs or images?
Yes! You can use wildcards and the end-of-string character: Disallow: /*.pdf$ will instruct search crawlers not to index any PDF documents on your website.
What is the difference between "Disallow: /" and "Disallow:"?
Disallow: / (with a slash) blocks crawlers from accessing your entire website starting from the root. Disallow: (empty with no slash) means "disallow nothing", granting complete unrestricted access to all pages.
How do I test whether my robots.txt file contains errors?
You can test your file using Google Search Console's official Robots.txt Tester tool, or run Rankzio’s free 74-Point AdSense Readiness Checker which inspects whether your robots.txt file is syntactically sound and accessible to search bots.
Can I have multiple Sitemap directives in a single robots.txt file?
Yes. If you manage multiple sitemap files (e.g., a post sitemap and a page sitemap), you can list each one on its own line: Sitemap: https://yourdomain.com/sitemap-posts.xml and Sitemap: https://yourdomain.com/sitemap-pages.xml.
Do AI crawlers like GPTBot and CCBot respect robots.txt?
Major AI organizations (including OpenAI, Anthropic, and Common Crawl) officially respect robots.txt exclusions. You can block them specifically: User-agent: GPTBot \n Disallow: / without affecting standard Google search indexing.
Does robots.txt affect subdomains?
Robots.txt applies strictly to the exact host domain where it is placed. A robots.txt file located on https://yourdomain.com/robots.txt does NOT control subdomains like https://blog.yourdomain.com/ or https://shop.yourdomain.com/. Each subdomain requires its own robots.txt file.