Like a lot of you out there, I’m one of those internet marketers that wears a million and one different hats. Almost every day I spend some duration of time on email marketing, PPC, website updates, search engine optimization and more. But more recently, SEO has become a much higher priority for my company, and subsequently, for me. After five years or so of updating meta tags, adding structured data markup, and blogging for keywords without a ton of success, we finally took the leap and purchased a subscription to a web-based SEO program. What did we find? Errors – a lot of errors. But I’m no web developer. So fixing all these problems in order to increase our PageRank involved a lot of research; research that made me realize I had been doing one thing in particular wrong for all these years. Apparently, I didn’t have a handle, at all, on when to use a “noindex” meta tag, a “nofollow” meta tag, a “nofollow” link tag, or a disallow list in my robots.txt file.
First, let me explain the differences between these methods.
The robots.txt file tells search engines not to crawl the pages included. These pages can still accrue PageRank (link juice), and the file itself isn’t even a full proof way to exclude these pages from being indexed. It is still possible that they’ll be indexed and show up in search results. However, you’re really telling Google that these pages, files, etc. aren’t as important as the others on your site, so crawl the more important ones first. I typically put image files hosted on our sever in the robots.txt file. Considering our company’s product is intangible, there isn’t a need for any of our images to be indexed.
A page with a noindex meta tag, <meta name=”robots” content=”noindex”>, allows search engines to crawl the page and give it a PageRank, but it won’t be indexed and won’t show up in search results. Since these pages accumulate PageRank they still pass PageRank on to the other pages they link to.
There are two different ways to use the nofollow tag. The first is on the page level. It appears between the <head></head> section in your code as a meta tag, <meta name=”robots” content=”nofollow”>. This tells search engines not to follow any outgoing links on that page so that PageRank doesn’t flow from that page.
The second way to use the nofollow tag is on a specific link itself with rel=”nofollow”. This prevents PageRank from being transferred through that individual link but allows the rest of the links on that page to pass on PageRank.
Putting It All Together
You can use nofollow and noindex together so that a page is not indexed in Google’s search results and does not pass on PageRank. This is done in the <head></head> section of your code with the meta tag <meta name=”robots” content=”noindex, nofollow”>.
Keep in mind that more pages you have on your site, the more PageRank is distributed between all of them. By eliminating unnecessary pages from being indexed and ranked, you can increase your PageRank on your remaining, more important pages.
By now, hopefully you know when use to use noindex, nofollow, or a robots.txt file. Still have questions? Leave and comment and we’ll keep the conversation going!