Why is noindex more effective than robots.txt?
-
In this post, http://www.seomoz.org/blog/restricting-robot-access-for-improved-seo, it mentions that the noindex tag is more effective than using robots.txt for keeping URLs out of the index. Why is this?
-
Good answer, also we have seen that sometimes bots come directly to your site via a link and do not always visit the robots.txt file and will therefore index the first page they come to.
Matt Cutts has said before that the only 100% fail safe way of blocking search engines indexing something will be to have it password protected
-
The disallow in robots.txt will prevent the bots from crawling that page, but will not prevent the page from appearing on SERPs. If a page with a lot of links to it is disallowed in the robots.txt, it may still appear on SERPs. I've seen this on a few of my own pages... and Google picks a weird title for the page...
If you put the meta noindex tag, Google will actively remove that page from their search results when they re-index that page.
Here was one webmaster central thread I found about it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Search Results Pages Blocked in Robots.txt?
Hi I am reviewing our robots.txt file. I wondered if search results pages should be blocked from crawling? We currently have this in the file /searchterm* Is it a good thing for SEO?
Intermediate & Advanced SEO | | BeckyKey0 -
Effect of Publishing Blog Posts Resembling Classified Advertisements
Our site (www.nyc-officespace-leader.com) markets commercial real estate for lease in New York City. Any potential negative impact in terms of ranking and traffic by using our blog post in an unconventional manner? I am considering publishing a weekly post describing the latest commercial listings for lease. The post would be formatted and resemble classified advertising appearing in such newspapers as The New York Times. The ads are concise and appealing. Property listings drive a high click thru rate, so I believe blogs posts based on property listings and formatted like old newspaper ads might really improve visitor engagement. Each add could have a link to a corresponding listing page. Would using the blog in this manner every week have a detrimental effect or could prove beneficial? Thoughts??? lr6MIiR
Intermediate & Advanced SEO | | Kingalan10 -
Question about Syntax in Robots.txt
So if I want to block any URL from being indexed that contains a particular parameter what is the best way to put this in the robots.txt file? Currently I have-
Intermediate & Advanced SEO | | DRSearchEngOpt
Disallow: /attachment_id Where "attachment_id" is the parameter. Problem is I still see these URL's indexed and this has been in the robots now for over a month. I am wondering if I should just do Disallow: attachment_id or Disallow: attachment_id= but figured I would ask you guys first. Thanks!0 -
Reversing the bad effects of a problematic 301 redirect
I have a previously very strong ranking page that is now omitted from the SERPs, but only for one specific keyword phrase. I think I found the reason, which I'll explain, and I hope I can hear some confirmation of my theory and a way to correct it. Let's use the following made up domain and keywords: Political blog SiteA.com had a few news articles about "Blue Widgets" (like 10 out of 10,000 pages). They became exceedingly popular, so on SiteA.com we created a reference-type page about "Blue Widgets" and in the news articles we already had about Blue Widgets we added rich anchor text (Blue Widgets) links that pointed to this new About Blue Widgets page. (long before we wised up about keyword rich anchor texts and Google!) After seeing how much traffic was coming to the About Blue Widgets page, we created a whole new site, SiteB.com, which was about Widgets (not just Blue Widgets), a page for each color of widget, and other pages about widgets. SiteB.com has an important and popular page, SiteB.com/blue-widgets, which is about Blue Widgets. We then 301 redirected the SiteA.com's About Blue Widgets page to SiteB.com/blue-widgets. This page in SiteB.com ranked very high (like #2, #3) for years. Two weeks ago SiteB.com/blue-widgets fell out of the SERPs, but only for the phrase "Blue Widgets". The page still gets lots of traffic from other queries, and even the "Blue Widgets" query will bring up other pages on SiteB.com. So, the only thing hit is the specific query "Blue Widgets" for the specific page SiteB.com/blue-widgets. It seems obvious to me that Google took the combination of a) a site that it probably no longer liked since we sold it (SiteA.com) since it's gone downhill, b) the rich keyword anchor text on SiteA.com pages pointing to the SiteA.com page optimized for that keyword, and c) then being 301 Redirected to a SiteB.com Blue Widgets page optimized for that same anchor text. I only discovered the SiteA.com redirects last week, which I had completely forgotten about, and had them removed right away. My question is, 1) if this indeed was the issue, now that the redirects from SiteA.com to SiteB.com are gone will my ranking eventually go back to normal? and 2) is there anything I can do to get Google to notice the change and have it go back to how it was?
Intermediate & Advanced SEO | | bizzer0 -
ECommerce search results to noindex?
Hi, To avoid duplicated content and the possibility of thousands additional pages to an ecommerce website would it be a reasonable solution to have the page as a no-index, would this benefit the site? Thanks **Lantec **
Intermediate & Advanced SEO | | Lantec0 -
Is our robots.txt file correct?
Could you please review our robots.txt file and let me know if this is correct. www.faithology.com/robots.txt Thank you!
Intermediate & Advanced SEO | | BMPIRE0 -
Does title text of homepage effect ranking of sub pages?
Question is pretty much summed up in the title. I realize that title text on a specific page can effect the ranking of that page. But what I'm getting a feeling of lately is that google uses the title text of your homepage to effect the ranking of the site on a whole. Thoughts?
Intermediate & Advanced SEO | | adriandg1 -
Lazy loading images effect image seo?
Im using a wordpress plugin to lazy load images so that the site speed is a lot faster. Will this mess up image seo? The code for the image looks like this (with the site taken out) [
Intermediate & Advanced SEO | | Gordian<noscript></p> <p> </p> <p>I see it has the 1x1.gif it loads to speed up page speed but does the fact the link is to the correct place make it ok?</p> <p>Thanks for letting me know</p></noscript>](http://.com/wp-content/uploads/2012/02/IMG_9477.jpg)
0