What reasons exist to use noindex / robots.txt?
-
Hi everyone. I realise this may appear to be a bit of an obtuse question, but that's only because it is an obtuse question. What I'm after is a cataloguing of opinion - what reasons have SEOs had to implement noindex or add pages to their robots.txt on the sites they manage?
-
Many reasons. You don't want the admin pages of your site indexed, for example. You may not want all of the search queries that people perform on your site search to be indexed. You don't want or need your cart checkout being indexed for an ecommerce site. You don't want a print version and a web version of the same document indexed, so you exclude the print version from being indexed. Your site is in development, and you don't want it being indexed before it is ready.
For robots.txt in particular, some search engines now respect wildcards and you can exclude some session IDs via robots.txt. OSCommerce is real bad about creating session IDs and getting those indexed, then you have tons of different URLs indexed for the same page.
http://www.cogentos.com/bloggers-guide-to-using-robotstxt-and-robots-meta-tags-to-optimise-indexing/ is a post that explains some of the reasons to use robots and no-index on a Wordpress site.
-
There are a couple that come to my mind when i used them working for an agency. I remember one client had some temporary pages that didn't want to get indexed, explaining certain problem with a product at that time. We wanted the page to be live, but didn't want the problems that the product was having to show up in the search engines since it was just temporary.
Also, pages that are targeting same keywords that you dont want to erase or redirect and instead want to keep them live but at the same time you dont want to compete with the other main page. You just block it to the search engines.
Hope this helps
-
I really should have worded my question better. I'll try again.
**What reasons do people have for not wanting their pages show on search results? **
I've got a few reasons of my own, but I'm interested in seeing if there's any I hadn't thought of.
-
For pages you don't want them to show up on search results. =P
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
News site and duplicate/cannibalism/old
Hey everybody. I am working with a handful of medical news websites and some of these sites are rather large. I am noticing/suspecting there is some cannibalism going on here. How do news sites deal with this? Do they noindex the old stuff? Update the old stuff instead of writing a new piece all together? Curious what to do!
On-Page Optimization | | HashtagHustler0 -
Using Schema markup for Feefo reviews
I am a little confused about whether or not it is ok to use Schema markup with reviews collected through Feefo. We use Feefo to collect reviews from our customers and these get displayed on our website. We get service ratings as well as product ratings through Feefo. My question is: Is it ok to use Schema markup for these? I would have thought they would fall under 3rd party reviews, but this article from the Feefo website seems to suggest that it would be ok to use markup in the way they recommend. Can anyone confirm how Google handles review markup like this? Thanks in advance!
On-Page Optimization | | ViviCa10 -
Keyword density or No. of Time keyword used
Now, I know that there is no set figure to be used here, whichever metric you are using and it will depend on the article and what is natural. However, lets suppose for a minute that we are taking a keyword in isolation, and I have a 2000 word article using the keyword 17 times and rank no. 3 in Google SERPS. The no. 1 slot uses the keyword 8 times but only has a 800 word article and only a B grade on the onpage ranker. Of course, there are off page factors as well, but just wondering what your thoughts are on whether you look at density or total keyword usage. It is easy to just write without think about keyword density or usage, but occasionally you end up using the keyword about 50 times, and it is then I have to actually think about it. Other articles I barely use the keyword because the article just writes itself and it works out fine, but these are generally shorter. With longer articles on my best converting pages, I can't help but think about it more and it ends up a little hit and miss.
On-Page Optimization | | TheWebMastercom1 -
www vs no-www duplicate which should I use
site is no-www I caught this in archives. Will this by my fix? Mike Davis Online Marketing Manager at McKesson May 22, 2013 Easy fix: in your .htaccess file, use this RewriteEngine On
On-Page Optimization | | touristips
RewriteCond %{HTTP_HOST} !^domain.com
RewriteRule (.*) http://domain.com/$1 [R=301,L] Remember to replace domain.com with your domain name.
Enjoy!0 -
Removing old URLs that are being used for my on page optimization?
Is there a way to remove old URL's that are still being used for my keywords for my on page optimization? They are giving me grades of F since they no longer exist and if I change the URL to the current one, the grade becomes an A, but they are still showing after the new crawl.
On-Page Optimization | | Dirty0 -
Does anyone have any opinions on whether to use a pipe | in your title tags? Or commas or just nothing at all?
I am trying to decided whether to use: BootsJeansandLeathers.com Mens Jeans Mens Boots Mens Leathers Mens Footwear OR BootsJeansandLeathers.com | Mens Jeans | Mens Boots | Mens Leathers | Mens Footwear as my title?
On-Page Optimization | | ebowdublin0 -
Do JSON feed / XML Feeds have any SEO benefits?
We are looking to use a 3rd party customer review provider and I have been told by their rep that it has a feature which allows us to display reviews placed with them about us on our website via JSON feed or XML feed (I have limited technical knowledge so have no idea what this is but am hoping our developer does!) I wanted to ask whether providing data in either format has any SEO benefits? My initial thoughts to provide customer reviews was to be useful to the customer and be useful for search engines as it will be continually updating user generated content (?) Any help on this would be greatly appreciated.
On-Page Optimization | | jannkuzel0