Meta Robots "Index, Follow"
-
In my MozBar under "General Attributes" it says "index, follow" next to Meta Roberts for one of our client's websites. I've never seen "index, follow" before. I've seen it say "not found." What does index, follow mean and is that a bad thing? I know the reason should be obvious but this site has had a lot of problems and I'm wondering if this is related.
-
Hello!
Index/Follow means that you are allowing search engines to index your pages and follow links through those pages. If you didn't have this tag on the page, search engines would do it anyway. Basically, this meta tag isn't necessarily a bad thing and the search engines would be doing it even if the tag wasn't there.
You can use meta robots to do good things for your site, such as stop the crawling of blog archive pages to limit duplicate content. There's tons of resources here on MOZ about the robots file and I would suggest looking into it. This is a good one: http://moz.com/learn/seo/robotstxt
What type of problems is your site having?
-
Thank you! That was very helpful!
-
What you're seeing is the page's meta robots tag, which instructs the search engine bots to (in this case) index a specific page and follow links on the page. The opposite, "noindex,nofollow" instructs the search bots to not include this page in its index or follow links on the page. You can also use "noindex, follow" so the search engine keeps the page out of its index, but still follows the links on the page. I personally have never come across an example of "index, nofollow" but I imagine that it's possible - probably more of an outdated practice though related to hoarding links.
If you've never seen this attribute before, it may be because the page doesn't include the tag (in which case the bots, by default, assume "index, follow"). And as for whether or not having it is a bad thing - probably not, unless it's a page you really don't want visible in search results.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
"New" issues not previously found being shown?
I'm not sure what logic Moz is using for its reporting of Site Crawl issues, but it appears to be pretty flawed (unless I'm missing something, which is possible). I've got a client site that has been in Moz for about 6 months now. Every time the crawler runs, the same number of pages are reported as having been crawled. However I'm consistently getting "New Issues" reported that should have been reported during previous crawls. Example: A redirect chain was reported several month ago. The referring URL was the homepage of the website, and we tracked it down to an old link in the header. This was fixed, marked as resolved, and the issue was not shown on the next crawl. Several weeks later, the same issue was reported for a different page on the website - a page which has existed since 2014 and was already crawled many times. Again, we fixed. Fast-forward to the report that just ran on 12/1 and we have the same issue reported, for a different page, which has also existed for years and has been previously crawled. It's very hard to explain to a client "this item you are seeing has been resolved", only to have it continually crop back up in future reports. Note this is not limited to redirect chains - that's just an example. I'm seeing this for other items such as missing canonicals, duplicate titles, etc.
Moz Bar | | RucksackDigital0 -
Is the update site crawl feature following robot.txt rules?
I noticed that most of the errors would not be occurring if Moz's tool followed the rules implemented in sites robots.txt. Has anyone else seen this problem and do you know if Moz will fix this?
Moz Bar | | jamestown0 -
Moz is reporting weird email address URLs as 'Meta refresh' errors? Anything to worry about?
Under site crawl, Moz is reporting weird email address URLs as 'Meta refresh' errors. The URLs are: http://support@ihasco.co.uk and http://enquiries@ihasco.co.uk Once clicked, they redirect to our homepage. Anyone else ever had this? Is it anything to worry about? I don't think it is, but would be good to get some reassurance.
Moz Bar | | iHasco0 -
Need a link rel="canonical"??
I'm getting a Craw issue in my MOZ insights which says "It looks like you're not using the rel="canonical" META attribute on any of the pages we crawled" Do we need to have a rel="canonical" on our site? I can't find a reason to use it so don't know why it would be appearing as a crawl error? Thanks
Moz Bar | | O2C0 -
MozBar > General Attributes > Meta Robots > noindex
I'm having a hard time figuring out where the noindex value for Meta Robots is coming from so I can fix it. Can anybody spot the issue or point me to some docs that show were the MozBar finds this information http://www.produnkhoops.com
Moz Bar | | tatermarketing0 -
I'm getting, "you're not using the rel="canonical" META attribute" in my crawl diagnotic
I'm running a campaign crawler through Moz on this particular page: http://www.henley.ac.uk/executive-education/leadership-and-management-programmes/ but I'm getting a notifcaiton from Moz saying, "you're not using the rel="canonical" META attribute" I don't understand what this means!! Has anyone else had this problem, or can they help me understand what this means and how to fix it? Oh, and Happy Thanksgiving from the UK! Virginia
Moz Bar | | blackboxideas0 -
Moz just did my weekly "crawl" and claims my meta titles and descriptions are "missing," but they aren't. What's up with this?
I can check for my titles and descriptions in the source. They are there, always have been. what in the hell?
Moz Bar | | LifeisRosie0