Meta Robots "Index, Follow"
-
In my MozBar under "General Attributes" it says "index, follow" next to Meta Roberts for one of our client's websites. I've never seen "index, follow" before. I've seen it say "not found." What does index, follow mean and is that a bad thing? I know the reason should be obvious but this site has had a lot of problems and I'm wondering if this is related.
-
Hello!
Index/Follow means that you are allowing search engines to index your pages and follow links through those pages. If you didn't have this tag on the page, search engines would do it anyway. Basically, this meta tag isn't necessarily a bad thing and the search engines would be doing it even if the tag wasn't there.
You can use meta robots to do good things for your site, such as stop the crawling of blog archive pages to limit duplicate content. There's tons of resources here on MOZ about the robots file and I would suggest looking into it. This is a good one: http://moz.com/learn/seo/robotstxt
What type of problems is your site having?
-
Thank you! That was very helpful!
-
What you're seeing is the page's meta robots tag, which instructs the search engine bots to (in this case) index a specific page and follow links on the page. The opposite, "noindex,nofollow" instructs the search bots to not include this page in its index or follow links on the page. You can also use "noindex, follow" so the search engine keeps the page out of its index, but still follows the links on the page. I personally have never come across an example of "index, nofollow" but I imagine that it's possible - probably more of an outdated practice though related to hoarding links.
If you've never seen this attribute before, it may be because the page doesn't include the tag (in which case the bots, by default, assume "index, follow"). And as for whether or not having it is a bad thing - probably not, unless it's a page you really don't want visible in search results.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hald the keywords in my keyword list say "Gethering Metrics". How do I get that to complete?
I'vwe waited about three hours, with no change. Why does this happen — is it something I did? Will those metrics ever be gathered? Or do I need to re-do buildingthe list all over again? And how can I be sure it will be any dirfferent?
Moz Bar | | btreloar0 -
Moz says "Title Too Long", Yoast says title is the perfect length. Who's right?
For a bunch of my pages, the MOZ Crawl Report says "Title Too Long". Yoast on my site tells me that the titles are the correct length. How can these two things be at odds with each other? Which one is right?
Moz Bar | | TeamViviRealEstate0 -
How do can the crawler not access my robots.txt file but have 0 crawler issues?
So I'm getting this errorOur crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.https://www.evernote.com/l/ADOmJ5AG3A1OPZZ2wr_ETiU2dDrejywnZ8kHowever, Moz is saying I have 0 Crawler issues. Have I hit an edge case? What can I do to rectify this situation? I'm looking at my robots.txt file here: http://www.dateideas.net/robots.txt however, I don't see anything that woudl specifically get in the way.I'm trying to build a helpful resource from this domain, and getting zero organic traffic, and I have a sinking suspicion this might be the main culprit.I appreciate your help!Thanks! 🙂
Moz Bar | | will_l0 -
Annoying Popup saying "authentication required"! How do I resolve this?
I utilise the Moz bar quite rigourously as a content marketer but i'm having an annoying popup every second. Moz kept bugging me with a popup bar saying: "authentication required" asking me to input my username & password. Though filling it in correctly, it keeps bugging me every second. Sometimes even when I have disabled the toolbar it keeps poping up! How to resolve this?
Moz Bar | | iPrice_Marketing1 -
Will robots.txt override a Firewall for Rogerbot?
Hey everybody. Our server guy, who is sorta difficult, has put these ridiculous security measures in place which lock people out of our website all the time. Basically if I ping the website too many times I get locked out, and that's just on my own, doing general research. Regardless, all of our audits are coming back with 5xx errors and I asked if we could add rogerbot to the robots.txt. He seems to be resistant to the idea and just wants to adjust the settings to his firewall... Does anybody know if putting that in the Robots.txt will override his firewall/ping defense he has put in place? I personally think what he has done is WAY too overkill, but that is besides the point. Thanks everybody.
Moz Bar | | HashtagHustler0 -
How do you feel when Moz marks one of your questions as "answered?"
Hi everyone, This is not meant to be snarky at all, so I just want to preface my question with that. So, since the new re-branded Moz rolled out last year, I'm sure many of you have noticed that if you ask a question and it is answered by a Moz associate, your question is marked as "answered." I'm sorry, but I don't like this. Here's why, I'm the one who asked the question. I should be the one who determines if the answer was adequate for me, or if it didn't sufficiently answer my question. This is particularly true when my question doesn't have to do with a customer service issue or a Moz tool question. If I ask a question about SEO, Content, CRO, marketing or any other subject, I feel like it should be me and only me who determines whether or not I feel like my question is answered. In addition to this, Moz is actually depriving themselves of useful UGC by shutting down questions in this way. How? Because when the rest of us who frequent the Q & A see a question that's already been marked as "answered" we tend not to open it, read it and respond, because we think that person has already gotten what they needed....when in fact, it could be that a Moz associate has jumped in and marked their question as answered when it really wasn't. Consequently, we all miss out. I propose/move that Moz associates can only mark questions as "answered" when they pertain directly to Q & A about Moz tools, service and support. All other questions must be marked as "answered" only by the asker or closed as "answered" after they have been dormant for 6 months or more. Can I get a second (motion) ?
Moz Bar | | danatanseo4 -
Why is there no Mozscape index update?
There was supposed to be an update on August 26th. I see other posts asking with no Moz response. I am hoping if enough of us ask maybe someone at Moz will respond.
Moz Bar | | EcommerceSite2 -
Moz "Crawl Diagnostics" doesn't respect robots.txt
Hello, I've just had a new website crawled by the Moz bot. It's come back with thousands of errors saying things like: Duplicate content Overly dynamic URLs Duplicate Page Titles The duplicate content & URLs it's found are all blocked in the robots.txt so why am I seeing these errors?
Moz Bar | | Vitalized
Here's an example of some of the robots.txt that blocks things like dynamic URLs and directories (which Moz bot ignored): Disallow: /?mode=
Disallow: /?limit=
Disallow: /?dir=
Disallow: /?p=*&
Disallow: /?SID=
Disallow: /reviews/
Disallow: /home/ Many thanks for any info on this issue.0