What reasons exist to use noindex / robots.txt?
-
Hi everyone. I realise this may appear to be a bit of an obtuse question, but that's only because it is an obtuse question. What I'm after is a cataloguing of opinion - what reasons have SEOs had to implement noindex or add pages to their robots.txt on the sites they manage?
-
Many reasons. You don't want the admin pages of your site indexed, for example. You may not want all of the search queries that people perform on your site search to be indexed. You don't want or need your cart checkout being indexed for an ecommerce site. You don't want a print version and a web version of the same document indexed, so you exclude the print version from being indexed. Your site is in development, and you don't want it being indexed before it is ready.
For robots.txt in particular, some search engines now respect wildcards and you can exclude some session IDs via robots.txt. OSCommerce is real bad about creating session IDs and getting those indexed, then you have tons of different URLs indexed for the same page.
http://www.cogentos.com/bloggers-guide-to-using-robotstxt-and-robots-meta-tags-to-optimise-indexing/ is a post that explains some of the reasons to use robots and no-index on a Wordpress site.
-
There are a couple that come to my mind when i used them working for an agency. I remember one client had some temporary pages that didn't want to get indexed, explaining certain problem with a product at that time. We wanted the page to be live, but didn't want the problems that the product was having to show up in the search engines since it was just temporary.
Also, pages that are targeting same keywords that you dont want to erase or redirect and instead want to keep them live but at the same time you dont want to compete with the other main page. You just block it to the search engines.
Hope this helps
-
I really should have worded my question better.
I'll try again.
**What reasons do people have for not wanting their pages show on search results? **
I've got a few reasons of my own, but I'm interested in seeing if there's any I hadn't thought of.
-
For pages you don't want them to show up on search results. =P
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Using Canonical Tags on Every Page
I'm doing competitive research and noticed that one of our competitors (who outranks us) uses canonical tags on every page on their site. The canonical tags reference the page they are on. For example. www.competitor.com/product has a canonical tag of www.competitor.com/product. Does anyone use this practice? It seems strange to me. Thank you, Kristen
On-Page Optimization | | Ksink0 -
Removing navigation menu items/links on homepage
We are redesigning our website after a long stint with an SEO firm who also handled our design/dev. We want to clean up the links on our homepage but don't want to screw up our IA or SEO. We want to delete some navbar menu items and a whole bunch on random links to our evergreen content below the fold. Would we need to reposition those navbar items/content links to our footer or somewhere else on the homepage to maintain our internal linking structure? It would be great if you could take a look at our site and give us any suggestions or advice on the best way to go about this. Thanks!
On-Page Optimization | | Lorne_Marr1 -
Can we listed URL on Website sitemap page which are blocked by Robots.txt
Hi, I need your help here. I have a website, and few pages are created for country specific. (www.example.com/uk). I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page) I really appreciate your help. Thanks, Nilay
On-Page Optimization | | Internet-Marketing-Profs0 -
Blog.mysite.com or mysite.com/blog?
Hi, I'm just curious what the majority think of what's the best way to start a blog on your website for SEO benefits. Is it better to have it under a sub domain or a directory? Or does it even matter?
On-Page Optimization | | truckguy770 -
Duplicate content http:// something .com and http:// something .com/
Hi, I've just got a crawl report for a new wordpress blog with suffusion theme and yoast wordpress seo module and there is duplicate content for: http:// something .com and http:// something .com/ I just can't figure out how to handle this. Can I add a redirect for .com/ to .com in htaccess? Any help is appreciated! By the way, the tag value for rel canonical is **http:// something .com/ **for both.
On-Page Optimization | | DanielSndstrm0 -
How do i chaneg a url without losing pre-existing linkjuice?
my client has a page on his site: www.revisitors.com/REF/TEST the page is for a free traffic offer...i have this page currently ranking 25th or so for "free traffic" - a great keyword to rank for to promote this offer with..... i want the url to be www.revisitors.com/free-traffic.html and then change the title tags to free traffic | free targeted traffic or something along those lines....i have hardly put in any linkbuilding work at all to get to 25 - it was a surprise but now that we are there i want to tighten things up and optimize as well as possible.... how do i do this without losing previously built linkjuice and without having a duplicate content issue for having both pages exist with a 301 redirect.... if i am wrong about something please dont hesitate to set me straight...i am only masquerading as an seo expert these days anyway.....thanks.
On-Page Optimization | | Ezpro90 -
Customer Review Capture / Google Approved Review Sites
We are interested in showing customer reviews on our website (in UK for seo purposes- UGC) and are initially reluctant to use a 'What customer say' or Testimonial page as we think customers may think we have just made these reviews up? I wanted to ask what methods you folks use to capture reviews? If you use 3rd party providers do you have any recommendations? (I found this link but it seems a little outdated as it doesn't include for example eKomi:http://www.seomoz.org/blog/how-to-rank-well-in-google-products-search-a-big-list-of-places-to-get-reviews) Thanks in advance.
On-Page Optimization | | jannkuzel0 -
Correct use of Canonical link vs 301 redirect
Hi All, Seeking yet more advice. SEOMOZ tools have told me I have duplicate content on one of my sites and I am keen to clean this up. I am not to familiar with the following so thought I would ask. The duplicate content is shown on : www.mysite.com www.mysite.com/index.html Obviously I only see index.html when I check the code so what is the best method of resolving the duplicate content, Canonical or 301? Can you give me an example 🙂 Thanks all
On-Page Optimization | | wedmonds0