What is the X-Robots-Tag?
The X-Robots-Tag is an optional component of the HTTP response header that informs search engines how to crawl and index a web page. Unlike the meta robots tag, which is reserved for HTML pages, the X-Robots-Tag can be used for non-HTML files such as images, text files, and PDFs.
Here’s an example of an HTTP response with an X-Robots-Tag:
HTTP/1.1 200 OK
Date: Tue, 15 Nov 2022 11:38:17 GMT
Content-encoding: gzip
(...)
X-Robots-Tag: noindex
(...)
Why is the X-Robots-Tag Important?
The X-Robots-Tag is important because it allows you to control how search engines index and crawl not just HTML pages but other file types as well. This capability is particularly useful for managing large websites with a variety of content types. Key benefits include:
- Flexibility: Apply directives to non-HTML files such as PDFs, images, and videos.
- Global Application: Use the tag to apply rules site-wide or to specific directories and file types.
- Regular Expressions: Employ advanced patterns to target multiple URLs efficiently.
Common Directives for X-Robots-Tag
You can use any directive that is valid for the meta robots tag with the X-Robots-Tag. Here are some commonly used directives:
- noindex: Prevents the page or file from being indexed by search engines.
- nofollow: Prevents search engines from following links on the page.
- none: Equivalent to "noindex, nofollow."
- noarchive: Prevents search engines from showing a cached version of the page.
- nosnippet: Prevents search engines from showing a snippet of the page in search results.
For example, to instruct Googlebot not to index or follow any links in PDF files, you could use:
<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Files>
How to Set Up the X-Robots-Tag
On Apache Server
You can set up the X-Robots-Tag in the .htaccess file or httpd.conf file. Here’s how to apply a noindex directive to all PDF files:
<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex"
</Files>
On NGINX Server
For NGINX, you can add the directive in the site’s .conf file:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex";
}
Setting Directives for Specific Crawlers
You can specify directives for particular crawlers by naming them:
Header set X-Robots-Tag "googlebot: noindex, nofollow"
Viewing the X-Robots-Tag
Unlike the robots meta tag, the X-Robots-Tag is part of the HTTP response header and not visible in the HTML code. Here’s how to view it in Google Chrome:
- Load the URL in Google Chrome.
- Right-click on the page and select “Inspect” to open the developer tools.
- Select the “Network” tab.
- Reload the page.
- Select the relevant file on the left panel; the HTTP headers, including the X-Robots-Tag, will be displayed on the right panel.
FAQs
Can I use the X-Robots-Tag with HTML pages?
Yes, but it’s more commonly used for non-HTML files. For HTML pages, it’s easier to use the meta robots tag within the HTML code.
How do I test if my X-Robots-Tag is working?
Use the Inspect tool in your browser's developer tools to check the HTTP response headers. You can also use online tools like Google's URL Inspection Tool.
Can the X-Robots-Tag override robots.txt?
No, the X-Robots-Tag cannot override robots.txt directives. If a URL is disallowed in robots.txt, search engines will not crawl it and, consequently, won't see the X-Robots-Tag directives.
By understanding and properly implementing the X-Robots-Tag, you can gain finer control over how search engines interact with your website's various file types, enhancing your site's SEO and user experience.
