What Is a Meta Robots Tag?
A meta robots tag is an HTML tag that gives search engine crawlers instructions on how to index, crawl, and display a page in search results. It’s placed within the <head> section of the webpage.
Here’s an example in code:
<meta name="robots" content="noindex, nofollow">
The meta robots tag is mainly used to control page indexing and crawling and can be found in the page’s HTML code. For pages without an HTML head section or for resources like images or PDFs, the X-Robots tag is used instead.
Unlike the meta robots tag, the X-Robots tag is an HTTP header sent with the webpage’s response.
You can use the Ahrefs SEO Toolbar browser extension to check a page’s meta robots tag – The “indexability” section shows crawler directives based on both meta tags and HTTP headers.
It’s important to remember that search engine crawlers can only follow the instructions in the meta robots tag if the page is allowed to be crawled according to the rules in the robots.txt file. If you want a page to be excluded from search results, the page must be crawlable so that search engines can read the “noindex” directive.
Beyond controlling page indexing and crawling, the meta robots tag can also instruct search engines not to show a text snippet or video preview in search results or to avoid caching the page.
Importance of Meta Robots Tag
Webmasters often want to prevent certain pages from appearing in search engines, such as those with thin content, PPC landing pages, or admin and “thank you” pages. The meta robots tag is the most reliable way to achieve this by using the “noindex” directive.
Additionally, the “nofollow” attribute in the meta robots tag ensures that all links on the page are treated as “nofollow” by search engine crawlers like Googlebot, meaning no link equity or “link juice” is passed to the linked pages.
This tag can also control how a page is indexed and displayed to users – For example, a “nosnippet” directive can be added to stop search engines from showing a text snippet from the page in search results.
Setting Up the Meta Robots Tag
Setting up the meta robots tag in WordPress is straightforward with SEO plugins like Yoast and RankMath. Here’s how to do it:
Using Yoast SEO Plugin
- Navigate to the Advanced Section below the content editing blocks for your post or page.
- Look for the dropdown option labelled “Allow search engines to show this Post in search results?”
- This controls the “noindex” attribute for the page. Selecting “No” will apply the “noindex” directive, ensuring the page isn’t indexed by search engines.
Using the RankMath Plugin
To noindex pages with RankMath, go to the Advanced tab and select the “No Index” option.
