What Are Robot Meta Tags? And How to Implement them
Robots meta tags (or “meta robots tags”) are pieces of code that provide crawlers instructions for how to crawl or index web page content. Whereas robots.txt file tags give bots suggestions for how to crawl a website’s pages, robots meta tags provide more firm instructions on how to crawl and index a page’s content.
By using robots meta tags, website owners can prevent certain pages from being indexed, block search engines from following links, or specify how to display snippets in search results. This level of control is crucial for optimizing a website’s visibility and ensuring that only the most relevant content appears in search engine results.
There are two types of robots meta tags: those that are part of the HTML page (like the meta robotstag) and those that the web server sends as HTTP headers (such as x-robots-tag). The same parameters (i.e., the crawling or indexing instructions a meta tag provides, such as “noindex” and “nofollow” in the example above) can be used with both meta robots and the x-robots-tag; what differs is how those parameters are communicated to crawlers.
Meta tags give crawlers instructions about how to crawl and index information they find on a specific webpage. If these tags are discovered by bots, their parameters serve as strong suggestions for crawler indexation behavior. But as with robots.txt files, crawlers don’t have to follow your meta tags, so it’s a safe bet that some malicious web robots will ignore your tags.
Below are the parameters that search engine crawlers understand and follow when they’re used in robots meta tags. The parameters are not case-sensitive, but do note that it is possible some search engines may only follow a subset of these parameters or may treat some tags slightly differently.
Indexation-controlling parameters:
Below are the parameters that search engine crawlers understand and follow when they’re used in robots meta tags, also known as robots tags.
Noindex: Tells a search engine not to index a page.
Index: Tells a search engine to index a page. Note that you don’t need to add this meta tag; it’s the default.
Follow: Even if the page isn’t indexed, the crawler should follow all the links on a page and pass equity to the linked pages.
Nofollow: Tells a crawler not to follow any links on a page or pass along any link equity.
Noimageindex: Tells a crawler not to index any images on a page.
None: Equivalent to using both the noindex and nofollow tags simultaneously.
Noarchive: Search engines should not show a cached link to this page on a SERP.
Nocache: Same as noarchive, but only used by Internet Explorer and Firefox.
Nosnippet: Tells a search engine not to show a snippet of this page (i.e. meta description) of this page on a SERP.
Noodyp/noydir [OBSOLETE]: Prevents search engines from using a page’s DMOZ description as the SERP snippet for this page. However, DMOZ was retired in early 2017, making this tag obsolete.
- Unavailable_after: Search engines should no longer index this page after a particular date.
Types of robots meta tags
There are two main types of robots meta tags: the meta robots tags and the x-robots-tag. Any parameter that can be used in a meta robots tag can also be specified in an x-robots-tag.
We’ll talk about both the meta robots tags and x-robots tag tags below.
Meta robots tag
The meta robots tag, commonly known as “meta robots” or colloquially as a “robots tag,” is part of a web page’s HTML code and appears as code elements within a web page’s < head> section:
The meta robots tag applies specifically to the individual page containing it, unlike the robots.txt file which governs access across the entire website.
Code sample:
<meta name="robots" content="[PARAMETER]">While the general <meta name="robots" content="[PARAMETER]"> tag is standard, you can also provide tags to specific crawlers by replacing the "robots" with the name of a specific user-agent. For example, to target a directive specifically to Googlebot, you'd use the following code:
<meta name="googlebot" content="[DIRECTIVE]">Want to use more than one directive on a page? As long as they're targeted to the same "robot" (user-agent), multiple tags can be included in one meta directive – just separate them by commas. Here's an example:
<meta name="robots" content="noimageindex, nofollow, nosnippet">This tag would tell robots not to index any of the images on a page, follow any of the links, or show a snippet of the page when it appears on a SERP.
If you're using different meta robots tag tags for different search user-agents, you'll need to use separate tags for each bot.
Meta robots tag syntax
The meta robots tag syntax consists of two attributes: name and content. The name attribute specifies which crawler should follow the instructions, while the content attribute provides the instructions themselves. The most common values for the name attribute are “robots” (which applies to all crawlers) and specific crawler names like “googlebot” or “bingbot”. The content attribute can take various values, including “index”, “noindex”, “follow”, “nofollow”, and others. For example, a meta robots tag might look like this: <meta name="robots" content="noindex, nofollow">. This tag tells all search engine crawlers not to index the page and not to follow any links on it.
Examples of robots meta tags
Noindex and nofollow tags
The noindex and nofollow tags are two of the most commonly used values for the content attribute. Noindex tells search engines not to index a page, effectively keeping it out of search results. Nofollow instructs them not to follow links on a page, which means that the linked pages do not receive any link equity from the page. These tags can be used separately or in combination, depending on the desired outcome. For example, a page with a noindex directive will not be indexed, but search engines may still follow links on the page unless a nofollow directive is also present. Using both tags together ensures that the page is neither indexed nor does it pass any link equity.
X-robots-tag
While the meta robots tag allows you to control indexing behavior at the page level, the x-robots-tag can be included as part of the HTTP header to control indexing of a page as a whole, as well as very specific elements of a page.
While you can use the x-robots-tag to execute all of the same indexation directives as meta robots, the x-robots-tag directive offers significantly more flexibility and functionality that the meta robots tag does not. Specifically, the x-robots permits the use of regular expressions, executing crawl directives on non-HTML files, and applying parameters at a global level.
To use the x-robots-tag, you'll need to have access to either your website's header .php, .htaccess, or server access file. From there, add your specific server configuration's x-robots-tag markup, including any parameters. This article provides some great examples of what x-robots-tag markup looks like if you're using any of these three configurations.
Here are a few use cases for why you might employ the x-robots-tag:
- Controlling the indexation of content not written in HTML (like flash or video)
- Blocking indexation of a particular element of a page (like an image or video), but not of the entire page itself
- Controlling indexation if you don't have access to a page's HTML (specifically, to the <head> section) or if your site uses a global header that cannot be changed
- Adding rules to whether or not a page should be indexed (ex. If a user has commented over 20 times, index their profile page)
Implementing robots meta tags
Implementing robots meta tags is a straightforward process that involves adding the tag to the <head> section of an HTML page. The tag should be placed before the closing </head> tag and should include the name and content attributes. For example: <meta name="robots" content="noindex, nofollow">. This tag tells search engines not to index the page and not to follow any links on it. Website owners can also use plugins or modules to add robots meta tags to their pages, especially if they are using a content management system like WordPress or Joomla. These tools simplify the process and ensure that the tags are correctly implemented across the entire site.
Implementing X-robots-tag
The X-Robots-Tag is an HTTP header that serves the same purpose as the meta robots tag but is used for non-HTML files like images and PDFs. To implement the X-Robots-Tag, website owners need to add a specific line of code to their website’s configuration files, such as the .htaccess file for Apache servers or the .conf file for Nginx servers. The code should include the X-Robots-Tag directive and the desired values, such as “noindex” or “nofollow”. For example: Header set X-Robots-Tag "noindex, nofollow". This directive tells search engines not to index the file and not to follow any links within it. Using the X-Robots-Tag provides greater flexibility and control, especially for non-HTML content that cannot include meta tags directly.
SEO best practices with robots meta tags
- All meta tags (robots or otherwise) are discovered when a URL is crawled. This means that if a robots.txt file disallows the URL from crawling, any meta directive on a page (either in the HTML or the HTTP header) will not be seen and will, effectively, be ignored.
- In most cases, using a meta robots tag with parameters "noindex, follow" should be employed as a way to to restrict crawling or indexation instead of using robots.txt file disallows.
- It is important to note that malicious crawlers are likely to completely ignore meta tags and as such, this protocol does not make a good security mechanism. If you have private information that you don't want to make publicly searchable, choose a more secure approach, such as password protection, to keep visitors from viewing confidential pages.
You do not need to use both meta robots and the x-robots-tag on the same page – doing so would be redundant.
Updated by Chima Mmeje — November 18, 2024.
Improve your technical site health with Moz Pro
Keep tabs on critical crawler issues (and tons of other details) with the help of Moz Pro's Site Crawl. Take a 30-day free trial on us and see what you can achieve: