What is a Meta Robots Tag?
A meta robots tag is an HTML tag that provides specific instructions to search engine crawlers on how to index, crawl, and display a webpage in search results. It is placed within the <head> section of the webpage. The meta robots tag is used to control various aspects of how search engines handle the content on the page.
Here’s an example of how it looks in the code:
<meta name="robots" content="noindex, nofollow">
In this example:
noindextells search engines not to index the page.nofollowtells search engines not to follow any links on the page.
For resources that lack an HTML head section (e.g., images, PDFs), the X-Robots tag can be used. Unlike the meta robots tag, the X-Robots tag is an HTTP header sent with the response of a web page.
Verifying the Meta Robots Tag
You can verify the meta robots tag for a page using various tools and browser extensions, such as the Ranktracker SEO Toolbar, which provides directives for crawlers based on both the meta tags and HTTP headers.
Importance of Meta Robots Tags
Meta robots tags are important for several reasons:
- Control Indexing: Prevent certain pages from appearing in search results (e.g., thin content, admin pages, thank you pages).
- Control Crawling: Prevent search engines from following links on the page.
- Enhance User Experience: Control how snippets and previews are displayed in search results.
Rules for Meta Robots Tag
Here are the rules or directives that can be used with the meta robots tag:
- noindex: Instructs search engines not to index the page.
- nofollow: Instructs search engines not to follow any links on the page.
- none: Equivalent to noindex, nofollow.
- all: Equivalent to index, follow (default behavior).
- noarchive: Prevents search engines from showing a cached version of the page.
- nositelinkssearchbox: Prevents Google from showing a sitelinks search box in the search results.
- nosnippet: Prevents search engines from displaying a text snippet from the page in search results.
- indexifembedded: Instructs Google to index content embedded through iframes or similar HTML tags on a noindex page.
- max-snippet: Specifies the maximum number of characters in text snippets.
- max-image-preview: Specifies the size of image previews in search results.
- max-video-preview: Sets the maximum number of seconds for video snippets.
- notranslate: Instructs Google not to provide a translation for the web page.
- noimageindex: Tells Google not to index images on the page.
- unavailable_after: Instructs Google not to show a page in search results after a specified date/time.
For more detailed information on these rules, refer to Google’s documentation on the robots meta tag.
Implementing Meta Robots Tag
Implementing the meta robots tag in WordPress is straightforward using popular SEO plugins like Yoast and RankMath.
Using Yoast SEO Plugin:
- Go to the Advanced Section below the editing blocks for your post or page.
- The dropdown “Allow search engines to show this Post in search results?” controls the “noindex” attribute on the page.
Using RankMath Plugin:
- Open the Advanced tab while editing a post or page.
- Select the “No Index” option.
Conclusion
Meta robots tags are a powerful tool for webmasters to control how search engines interact with their websites. By implementing these tags correctly, you can improve your site's SEO, enhance user experience, and ensure that only the most relevant pages are indexed and displayed in search results.
