Check Your Robots txt and Meta Tag for noindex and nofollow
Make sure your Robots.txt file and/or your page content is properly using the noindex and nofollow parameters. In most cases you should not use the Robots Meta Tag or include a noindex or nofollow in your robots.txt file. Make sure you search and understand how this tag is being used on your pages or in your robots.txt.
I worked with a customer who complained about search traffic. Unfortunately they had accidently set a flag in their website design configuration which added a Meta Tag for Robots to noindex on all their website pages. When the search engine robots like Google bot see this Meta Tag, there algorithms will prevent these pages from showing up in their search results.
You can include this information for the Robots in either your robots.txt file in your root directory or within the <head> html for a specific page. It’s a best practice to specifically include it on pages where you either want a NOINDEX or NOFOLLOW. If you want robots to index and follow links on your page, don’t worry about adding this META TAG. Robots by default index and follow links on all pages.
Here is an example of the Meta tag for the HTML on your page:
<HR>
<META NAME=”ROBOTS” CONTENT=”NOINDEX, FOLLOW”>
or
<META NAME=”ROBOTS” CONTENT=”NOINDEX”>
The above stops the robots from indexing the content on this page but allow them to follow links. You may use either format since FOLLOW is the default and this parameter can be left off.
<HR>
<META NAME=”ROBOTS” CONTENT=”INDEX, NOFOLLOW”>
or
<META NAME=”ROBOTS” CONTENT=”NOFOLLOW”>
This will allow this page to be indexed in their search results but restricts them from following links.
<HR>
<META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>
If you haven’t figured it out by now, you should probably leave you SEO work to someone else.
<HR>
<META NAME=”ROBOTS” CONTENT=”INDEX, FOLLOW”>
Just to finish it off, you can actually write a META TAG for the Robots to follow links and index your pages. Again this TAG is almost never used because it’s the default for search engines.
Return to SEO home page
Tweet This Post
