What reasons exist to use noindex / robots.txt?
-
Hi everyone. I realise this may appear to be a bit of an obtuse question, but that's only because it is an obtuse question. What I'm after is a cataloguing of opinion - what reasons have SEOs had to implement noindex or add pages to their robots.txt on the sites they manage?
-
Many reasons. You don't want the admin pages of your site indexed, for example. You may not want all of the search queries that people perform on your site search to be indexed. You don't want or need your cart checkout being indexed for an ecommerce site. You don't want a print version and a web version of the same document indexed, so you exclude the print version from being indexed. Your site is in development, and you don't want it being indexed before it is ready.
For robots.txt in particular, some search engines now respect wildcards and you can exclude some session IDs via robots.txt. OSCommerce is real bad about creating session IDs and getting those indexed, then you have tons of different URLs indexed for the same page.
http://www.cogentos.com/bloggers-guide-to-using-robotstxt-and-robots-meta-tags-to-optimise-indexing/ is a post that explains some of the reasons to use robots and no-index on a Wordpress site.
-
There are a couple that come to my mind when i used them working for an agency. I remember one client had some temporary pages that didn't want to get indexed, explaining certain problem with a product at that time. We wanted the page to be live, but didn't want the problems that the product was having to show up in the search engines since it was just temporary.
Also, pages that are targeting same keywords that you dont want to erase or redirect and instead want to keep them live but at the same time you dont want to compete with the other main page. You just block it to the search engines.
Hope this helps
-
I really should have worded my question better.
I'll try again.
**What reasons do people have for not wanting their pages show on search results? **
I've got a few reasons of my own, but I'm interested in seeing if there's any I hadn't thought of.
-
For pages you don't want them to show up on search results. =P
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Using a hyphen in title tags and the impact of spaces
Hi I am trying to arrive at a best practice template for a title tag for my organization so does the following template still holds Primary Keyword - Secondary Keyword | Brand Name will anything be impacted if I eliminate the spaces between the hyphen, will search bots be still able to treat the first one as a priority and the second as the secondary? Primary Keyword-Secondary Keyword | Brand Name thank you
On-Page Optimization | | lina_digital0 -
Should stop words be used in titles? I'm aiming for natural title structure.
I have fully optimized on-page SEO for the following keyword (not really the one I use, but it can serve as an example): -personal driver in los angeles Even though "in" is a stop word, I prefer to have a natural (non-robotic) structure for the title - I do this by including "in". I believe that "personal driver los angeles" is too spammy and too robotic. Is this a good or a bad thing?
On-Page Optimization | | zorsto0 -
How are your "Service Area" pages handling Penguin/Panda?
We just got a new client because of recent Penguin/Panda changes. A national "SEO" firm decided it was a good idea to set up a page for each service town or county they serve with nothing but duplicate content. Needless to say, on the week of the 23rd, their rankings tanked from 1st page (it's not a competitive niche) to 4th. I'm not bringing this up to brag, but rather because it got me thinking... How are your geographically targeted "service area" pages doing? Have the recent changes caused you to rethink your geographic targeting in any way?
On-Page Optimization | | BedeFahey0 -
A / B Split testing
If you have developed a server-side A / B split testing tool. How do you avoid that Google will think you are doing cloaking. When a user enters www.xxxxx.com/xxxxxx/ he gets either version 1 or version 2 back from the server on the same url. If the user does not accept session cookies he will always get version 1.
On-Page Optimization | | Stener0 -
How to best use our blog posts for SEO?
My company recently created a WordPress hosted blog. It is hosted completely separate from our company site. The primary domain for the blog is blog.mydomainname.com, but we immediately created a folder within our company website for www.mydomainname.com/blog that has a reverse proxy to the blog itself. I'm curious though if we should consider taking the content from the blog posts and re-creating that within our company website as well? The blog posts are very good SEO rich content, and we always struggle to find new content to put on our company website as it is already. Would like to get some folks thoughts on this. Thanks!
On-Page Optimization | | KHCreative0 -
Using magentos own url re-writes
We are changing ecommerce platforms. Is it best to use magentos own url re-writes to redirect every page of a site from its old url to it new one?
On-Page Optimization | | LadyApollo0 -
The Better Title to Use?
Hello Mozers- I am targeting the keywords "liposuction scottsdale", "liposuction phoenix", "liposuction mesa", "liposuction arizona". Out of the following two Titles below, one would you consider the better one? And Why? I am leaning towards the second example. If you have any more ideas that would be great. 1). Tummy Tuck Scottsdale, Phoenix, Mesa, Arizona 2).Tummy Tuck Scottsdale | Tummy Tuck Phoenix, Mesa, Arizona. Thanks!!
On-Page Optimization | | Red_Spot_Interactive0 -
Different names for Twitter/Facebook, and site?
Greetings, I want to optimize my site, www.atstrust.com. I know that posts to Facebook and Twitter help improve ranks. Does it matter though if my twitter/facebook sites have a different name than the domain I am optimizing? For example, my Facebook site is facebook.com/easyinspection. I assume that doesn't matter as long as the URLs on the site refer to my domain? Thanks! Eric
On-Page Optimization | | Ericc220