Every page in the world of websites has a Head section and the <head> section is made of Metas. Sometimes we want to talk to the Search Engine whether to opt-in or opt-out indexing a page.
Here's an example:
<html> <head> <meta name="robots" content="noindex,nofollow"> </head> <body> Hello </body> </html>
In the example above, we have defined in the page's head that the Search Engines should not index the page and should not follow the links in the page.
This is how we control what pages to show up in search results (SERPs).
What is Meta Robots Noindex and Nofollow?
The Robots Meta in the head section of an HTML file is there to tell the search engines to index and follow the given page or not.
The possible values for the Meta Robots are:
<meta name="robots" content="index,follow" />
<meta name="robots" content="noindex,nofollow" />
<meta name="robots" content="noindex,follow" />
<meta name="robots" content="index,nofollow" />(Additionally, you can have "noarchive", "none", "all", "notranslate", etc. The whole list of possible values is too long and you may not need to know all of them. But if you're curious, here you can read Google's documentation for robots meta.)
Basically Noindex means: "Hey, Search Engine! Do not include this page in your search database" and nofollow means "do not follow the links you see in this page!"
By default, Search Engines will want to scan and index all of your pages that they can find. So, using "index" or "follow" in the Meta Robots is unnecessary. However, some developers prefer to put:
<meta name="robots" content="index,follow" />
in the head just to be sure that the SE will index the page.
But, don't do that. Seriously!
Why Should someone use noindex or nofollow, you may ask!
Here's why:
Some pages have no useful information, however you need them. For example let's say you have a "Thank You" page which you want to show after a user completes the contact form. Now, this is a page that no one would want to find in the search results when they search your name or keywords.
This is only for showing your gratitude after they decide to contact you. So, it makes sense to use Meta Robots Noindex in the header. Just to discourage the search engines from indexing that page.
But will the search engines really follow your instructions?
Well, not all the time! Actually they're free to index the pages if they find them useful even though we have discouraged them. For Google and Bing you can be 100% sure that they will not index the pages with Robots Meta Noindex declared. But other search engines may index if they want to.
Also, the search engines will probably read the noindex pages for statistics or data mining purposes, whether we like it or not.
Landing Pages vs. Noindex pages
Landing pages in SEO are the most important pages that we want the Search Engines definitely index and promote in their search results. However, by definition, any page that is shown in the search results that the users will click and land on our website is considered a Landing Page.
And unfortunately, not always we like the landing page that the Search Engine chooses for us!
As a result, we may want to consider noindexing the pages that we don't want to be a landing page. It makes total sense!
Let's say you have a category page page in a WordPress website. And that category page is aweful! No proper design, no optimization, no useful content.
It's just good for listing the blog posts or products that fall under that category.
You would never want that page to be the landing page for your target keywords. But the Search Engine likes to play games! :)
So, we can add the Noindex in the Robots Meta and tell Google: Hey, I know you like to play games, but the game is over. We don't want this category page to be shown to the users. Good boy!
And when you do that, if there was an improper landing page in the search results, they will dissappear from SERPs. It may harm your traffic for a while, but later, a better page will be chosen for your target keywords, when you carry on with a proper backlink strategy and on-page optimization.
What's the difference between Robots.txt and Robots Meta?
Well, as the name suggests, both are addressing the bots who scan or crawl the pages whether or not to read our website.
Robots.txt accepts some wildcard rules for whitelisting or blacklisting the bots. But as you know, the bots can obey or just ignore the rules defined in robots.txt.
And another difference is that you only have one robots.txt which is put in the root of your website's directory. For example, we can have https://mywebsite.com/robots.txt
And in that text file we can write something like this:
User-agent: *
Disallow:
Sitemap: https://mywebsite.com/sitemap.xmlThis is telling us for any user-agent (or bot), the disallowed pages are none. There is no page listed under "Disallow". Therefore all pages are accessible for all bots and user agents. And under that there's a definition of the sitemap location which some bots prefer to use instead of trying to find your pages manually.
So, both methods can be used to encourage or discourage search engines for indexing your pages.
Are noindex pages bad for SEO?
Noindex pages can be bad for SEO. If your important pages or even worse, if the whole website has Meta Robots Noindex, Google will not index any of your pages. And if you had organic traffic in the past, after your pages are noindexed, you will lose that traffic and no one will be able to find your noindex pages in the search results afterwards.
This is very important for Website owners! Because their website may be somehow noindexed and they may not be aware. So, the first thing the SEO Company should do is to check which pages have Noindex in the Meta Robots. And make sure none of the important pages are noindexed.
Are nofollow pages bad for SEO?
Just like Noindex pages, Nofollow declaration in the Meta Robots can harm your SEO Rankings. But this one has less effect. Because Google usually finds new pages in your websites using different methods. If a link is nofollowed in one page, it may be followed from another page. Or perhaps you have a sitemap which will give the Search Engine the list of all of your important pages so it won't matter if one page has nofollow links.
You may already know that the <a> tag in HTML can have "rel" attribute. And rel can be nofollow:
<a href rel="nofollow">Click Me!</a>Adding Robots Meta Nofollow will be equal to adding rel="nofollow" to all the links in the page.
Some people also use "noopener" or "noreferrer" to the rel attribute of the anchor links, to prevent the destination page have any idea where the user came from. But it's a bit off-topic for this article.
But as for nofollow attribute, it's recommended to use it on all outgoing links. External links better be nofollowed. Because your SEO rankings will drop if you have too many outbound links in a page. Nofollow will neutralize that effect.
And if you're adding Meta Robots Nofollow, Noindex to specific pages like Contact Us, Thank You page, etc., it will be fine for SEO.
How to manage Nofollow / Noindex pages for SEO?
As a SEO Expert, you should use SEO Tools to scan your client's website for Optimization Errors. For the start, you may want to see which pages have Robots Meta Nofollow or Noindex.
One thing you can do is to install Google Chrome extension or Firefox Addon named Nofollow.
Install Nofollow Chrome Extension
You can find this extention here for Chrome and here for Firefox.
It's a super-easy extension to use. After install, it will automatically highlight the pages with a small pop up window in the bottom right corner that alerts the SEO Expert if the opened page is Noindexed or Nofollowed.
And for scanning the whole site, we recommend using our amazing Niotik SEO Panel. It's a great SEO Tool for detecting on-page optimization problems as well as managing the whole SEO procedures.
If you're a SEO specialist and have lots of SEO projects, you must use Niotik SEO Panel, and it's free for most of the functions!
Conclusion
If you're an expert in SEO, you can use the power of Robots Meta and Robots.txt to control which pages of your target website will be indexed on Search Engines. If you're a website owner and have no idea how Search Engine or SEO works, if your pages have Meta Robots Noindex and you are not sure if they're correct, if your pages are not being indexed or wrong pages are shown in search results, you may need to contact a SEO Company to take care of your website.
