What reasons exist to use noindex / robots.txt?
-
Hi everyone. I realise this may appear to be a bit of an obtuse question, but that's only because it is an obtuse question. What I'm after is a cataloguing of opinion - what reasons have SEOs had to implement noindex or add pages to their robots.txt on the sites they manage?
-
Many reasons. You don't want the admin pages of your site indexed, for example. You may not want all of the search queries that people perform on your site search to be indexed. You don't want or need your cart checkout being indexed for an ecommerce site. You don't want a print version and a web version of the same document indexed, so you exclude the print version from being indexed. Your site is in development, and you don't want it being indexed before it is ready.
For robots.txt in particular, some search engines now respect wildcards and you can exclude some session IDs via robots.txt. OSCommerce is real bad about creating session IDs and getting those indexed, then you have tons of different URLs indexed for the same page.
http://www.cogentos.com/bloggers-guide-to-using-robotstxt-and-robots-meta-tags-to-optimise-indexing/ is a post that explains some of the reasons to use robots and no-index on a Wordpress site.
-
There are a couple that come to my mind when i used them working for an agency. I remember one client had some temporary pages that didn't want to get indexed, explaining certain problem with a product at that time. We wanted the page to be live, but didn't want the problems that the product was having to show up in the search engines since it was just temporary.
Also, pages that are targeting same keywords that you dont want to erase or redirect and instead want to keep them live but at the same time you dont want to compete with the other main page. You just block it to the search engines.
Hope this helps
-
I really should have worded my question better. I'll try again.
**What reasons do people have for not wanting their pages show on search results? **
I've got a few reasons of my own, but I'm interested in seeing if there's any I hadn't thought of.
-
For pages you don't want them to show up on search results. =P
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What Are other people using in replacement for sliders?
Hello, Moz Community! I am currently trying to replace a slider on our client's site. Sliders, in my opinion, are awful, they slow load times and just don't convey a solid message. I am using Wordpress and the visual composer plugin. Any ideas are really appreciated even they may seem a bit much, if I don't know how to do it I will figure it out. I apologize as well if this isn't the appropriate place for this type of question.
On-Page Optimization | | Striventa0 -
What is the best use of canonical ref on home (default/index) page?
Moz reports show duplicate content for my www.domain.com and www.domain.com/default.html How do I format the canonical ref to put on the default.html page?
On-Page Optimization | | bhsiao0 -
Best way to get the fastest WordPress site with existing template
I have a WordPress website that I started building for free and eventually paid the $50 bucks for the premium version. My goal right now is to increase the speed. The current speed rating for the site on Google Insights is 60/100 for mobile and 66/100 for desktop. I have used some plugins to help me increase the speed, but the previous number were the best I could do. How can I get this site at least into the 90/100 range? Should I buy another template, or can I hire someone to go through and optimize it. I would like the best option that is as cost effective as possible as this is a hobby site for me. Thanks!
On-Page Optimization | | FierceFrame1 -
Would it be inappropriate use of authorship mark-up for a directory profile?
If a business creates a profile in a directory and adds their Google+ page, and they add the domain/profile URL as a publisher they contribute to, is that a legit use? They are the author of the business profile, but it's not an article.
On-Page Optimization | | designquotes0 -
Is there a set length / restriction on ALT tag content?
Something in which is an essential part of any site but I cant for the life of me remember if there is a set / recommended limit to the size if should be or is restricted to.
On-Page Optimization | | SamPenno0 -
Site structure suggestions/feedback
I asked this on Reddit and got some some decent answers. I'm curious to see what the pro's of SEOmoz think. I've got a lead generation site for forklift parts--liftxparts.com. You can think of it similar to car parts, where we have sections for specific brands (e.x. Toyota forklift parts) and sections for specific categories (e.x. forklift filters). Right now, the site is structured in two main levels: the top level is a dozen or so brands (separate pages for Toyota forklift parts, Clark forklift parts, etc), and then the second level is the categories (separate pages for a dozen or so different categories like forklift filters, forklift engine parts, etc.).If you check out one of the pages, like Clark forklift parts for example (our top landing page)--liftxparts.com/clark-forklift-parts.html, you'll see that on the brand pages (they're all structured the same), we list all the different categories (with links to the same second level category pages) and "search" buttons. All pages point to the same lead capture form.This has been working pretty well--about 90% of visitors end up on our lead capture form, and a high percentage of those convert. We're working on increasing organic traffic now and I'm thinking our structure could use some improvement.Looking at the analytics, there are a lot more impressions for keywords like "clark forklift" than "clark forklift parts". One gap I've uncovered is while our average position, and by extension CTR and traffic, for phrases like "clark forklift parts" is quite good, it's not so good for broader and higher searched terms like "clark forklift". Should we add another level of hierarchy targeted to just general brands? So now we have content for clark forklift parts, but should we add a page for terms like "clark forklift"? Or should we just add some broader content to the existing brand pages? The pages are quite long already, I'm afraid adding more content to the bottom of the page isn't very functional. Our thinking is that we can increase average position for higher searched terms by adding content targeted to those terms. The question is how exactly to go about it and how to work it into our current site structure? Any feedback related to our site structure or even just related ideas about other ways to approach our goal of increasing organic traffic would be very much appreciated! Thanks!
On-Page Optimization | | wisamabdulla0 -
Using meta robots 'noindex'
Alright, so I would consider myself a beginner at SEO. I've been doing merchandising and marketing for Ecommerce sites for about a year and a half now and am just now starting to attempt to apply some intermediate SEO techniques to the sites I work on so bear with me. We are currently redoing the homepage of our site and I am evaluating what links to have on it. I don't want to lose precious link juice to pages that don't need it, but there are certain pages that we need to have on the homepage that people just won't search for. My question is would it be a good move to add the meta robots 'noindex' tag to these pages? Is my understanding correct that if the only link on the page is back to the homepage it will pass back the linkjuice? Also, how many homepage links are too many? We have a fairly large ecommerce site with a lot of categories we'd like to feature, but don't want to overdo the homepage. I appreciate any help!
On-Page Optimization | | ClaytonKendall0 -
Related keywords in title/H1 tag
Hi, I am trying to improve our rankings for pages with photos/images. For the title is it benificial to include keywords that are almost identical in nature? For example: "Brad Pitt Photos and Images" In Google trends photos and images are both commonly used words so including both seems like it would help. When I search for each one separately in Google (Brad Pitt Photos vs Brad Pitt Images) different sites are returned (except for the ones that include both image and photos keywords). I had read that Google knows that Images and Photos mean the same thing, but the search results do vary. I know stuffing all related combinations isn't good, but selective phrases seem to make a difference. Just want to verify if this makes sense. Thanks
On-Page Optimization | | NicB10