No Index Checker
Easily check whether your pages are blocked from search engine indexing with our No Index Checker. Ensure your site’s visibility and privacy are exactly as intended.
What’s a meta robots tag?
A piece of HTML in the section of a webpage, the meta robots tag gives search engines explicit instructions on how to crawl and index that page. This includes instructions on indexing and whether to appear in search results (i.e., “index” or “noindex”), whether links should be followed (i.e., “follow” or “nofollow”), and how to display content in search results (e.g., “max-snippet:150”).
Think of this tag as a way to guide search engines and control which pages get indexed. It’s a handy tool for ensuring that your important content is visible while low-value or duplicate pages are excluded from search results.
How to use the meta robots tag
The meta robots tag is inserted in the section of the HTML on a specific page. As this tag instructs search engines, you can use it to “index” or “noindex” a page to allow or prevent it from appearing in search results, and use “follow” or “nofollow” to control whether its links are followed by search engines.
Combine these directives to meet your SEO goals, for example, block a page from search results while still letting it pass link equity. Inserting the meta robots correctly and using the right directives ensures your pages are crawled and indexed as intended.
What are HTTP Headers?
HTTP headers like noarchive, nosnippet, noimageindex, and nocache help control how search engines treat your pages. Noarchive stops cached copies, nosnippet hides descriptions in search results, and noimageindex keeps images out of the index. Nocache prevents storing old versions of a page. Set through X-Robots-Tag headers or meta tags, these tools give you simple control over what shows up online.
Meta robots tags best practice
- Place the meta robots tag in the section of your HTML pages so search engines can read it right away.
- Use the “noindex” directive only for pages you don’t want to appear in search results, such as duplicates, low-value pages, or private content.
- If you want search engines to follow the links on a page but avoid indexing the page itself, combine “noindex” with “follow.”
- To avoid confusing search engines, make sure the meta robots tag doesn’t conflict with canonical tags or other directives.
- When redirecting, point directly to the final URL and avoid creating unnecessary redirect chains or hops.
Frequently Asked Questions
Where to place the meta robots tag?
Always place meta robots tags in the section of a webpage. As search engine crawlers process the first, placement here is important. If a tag is placed outside the or in the body of the page, search engines may ignore it, and your webpages may not appear as intended in search results.
What instructions can a meta robots tag give?
The meta robots tag allows you to control how search engines crawl and index a page. Common directives include:
index / noindex – Determines whether the page should be included in search engine indexes. Use “index” to allow indexing and “noindex” to prevent it.
follow / nofollow – Controls whether search engines should follow links on the page. “follow” allows link equity to pass to other pages, while “nofollow” prevents it.
noarchive – Prevents search engines from storing a cached copy of the page.
nosnippet – Stops search engines from showing a snippet or description in search results.
noimageindex – Prevents images on the page from being indexed by search engines.
nocache – Prevents caching. This tag is mostly used in older implementations.
“noindex” and canonical tags – What’s the difference?
A “noindex” tag tells search engines not to include a page in search results, while canonical tags indicate the preferred version of a page for ranking purposes. A canonical doesn’t prevent a page from being indexed. However, if it points to a “noindex” page, then the original page may also be excluded from search results.
How often should webpages be checked for indexability?
Review the indexability status of your webpages whenever new content is published, the site structure changes or SEO updates are implemented. Checking indexability regularly helps ensure that your important pages are indexable and that search engines can always access the content that you want visible.
