What reasons exist to use noindex / robots.txt?
-
Hi everyone. I realise this may appear to be a bit of an obtuse question, but that's only because it is an obtuse question. What I'm after is a cataloguing of opinion - what reasons have SEOs had to implement noindex or add pages to their robots.txt on the sites they manage?
-
Many reasons. You don't want the admin pages of your site indexed, for example. You may not want all of the search queries that people perform on your site search to be indexed. You don't want or need your cart checkout being indexed for an ecommerce site. You don't want a print version and a web version of the same document indexed, so you exclude the print version from being indexed. Your site is in development, and you don't want it being indexed before it is ready.
For robots.txt in particular, some search engines now respect wildcards and you can exclude some session IDs via robots.txt. OSCommerce is real bad about creating session IDs and getting those indexed, then you have tons of different URLs indexed for the same page.
http://www.cogentos.com/bloggers-guide-to-using-robotstxt-and-robots-meta-tags-to-optimise-indexing/ is a post that explains some of the reasons to use robots and no-index on a Wordpress site.
-
There are a couple that come to my mind when i used them working for an agency. I remember one client had some temporary pages that didn't want to get indexed, explaining certain problem with a product at that time. We wanted the page to be live, but didn't want the problems that the product was having to show up in the search engines since it was just temporary.
Also, pages that are targeting same keywords that you dont want to erase or redirect and instead want to keep them live but at the same time you dont want to compete with the other main page. You just block it to the search engines.
Hope this helps
-
I really should have worded my question better. I'll try again.
**What reasons do people have for not wanting their pages show on search results? **
I've got a few reasons of my own, but I'm interested in seeing if there's any I hadn't thought of.
-
For pages you don't want them to show up on search results. =P
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is using a H1 tag in a logo image bad for SEO?
We have brand logos on certain pages that have H1 tags in them - the H1 text being the brand's name, as this is what we'd want the title of the page to be. The logos are at the top of the page instead of a written title. But is this the best option for SEO? Do search engines value H1 tags in images as highly as a standard H1 tag?Would it be better for SEO to add an alt tag to the logo and add a separate H1 tag on the page that's also the name of the brand?
On-Page Optimization | | DVLighting0 -
HTTPS and HTTP both exist! How to handle?
I was asked to do some SEO work for a website and learned that just 6 weeks ago, their webmaster added an HTTPS instance of the site. Their backlinks all point to HTTP and the 6 pages that are already ranking are all on the HTTP site. I'm afraid to rock the boat by redirecting the site from HTTP to HTTPS as we may lose rank. What are some suggestions? If I just pull down the HTTPS will that hurt us? Would you just go ahead and redirect it? IF so, would you do each page individually or as a whole?
On-Page Optimization | | dk80 -
Is it safe to use Microformats and Schema.org on one page?
I'm running a news blog and I'm using schema.org/NewsArticle. I would like to know if I should add microformats on the article page too or just use the schema? If I add both, will it effect my sites SEO/ranking? Thank you
On-Page Optimization | | hakhan2010 -
What does Appropriate Use of Rel Canonical mean?
I'm a newb and see this on my report and I don't understand what it means. Any help?
On-Page Optimization | | smartapps0 -
Authorship - How bad is it to use a fictional character instead of the actual author ??
Hi I know authorship is all about the author hence should be a real person (the author) but in the case of children's books etc etc is it really not ok to develop authorship around the character rather the actual author ? For example kids searching google will recognise the character but not the author Even if G doesnt like it how bad is it to ignore them and proceed with the character ? All Best Dan
On-Page Optimization | | Dan-Lawrence0 -
Can I use the same text in my meta description as I put in my post excerpt?
Hi, I'm just trying to understand the right way to optimise my blog posts and this is likely a dumb question... but to what extent should the text in my meta description differ from the text in my post excerpts? cheers, Andrew
On-Page Optimization | | seowhiskey0 -
Should you use long tail keyword phrases in page names
for example if I was trying to rank for "spokane furnace repair" is it good or bad to name the page something like "spokane-furnace-repair.html"
On-Page Optimization | | Superflys0 -
Blocked By Meta Robots
Hi I logged in the other day to find that over night I received 8347 notices saying certain pages are being kept out of the search engine indexes by meta-robots. I have not changed my robots.txt in years and I certainly didn't block Google from visiting those pages. Is this a fault on Roger Mozbot behalf? Or is there a bot preventing 8000+ pages being indexed? Is there a way to find out what meta-robot is doing this and where? And how I can get rid of it? I usually rank between #3 and #5 for the term 'sex toys' on google.com.au, but I now rank #7 to #9 so it would seem some of my pages/content is being blocked. My website is www.theloveshop.com THIS IS AN ADULT TOYS SITE. There is no porn videos or anything like that on it, but just in case you don't wish to look at sex toys or are around kids I thought I would mention it. Blake
On-Page Optimization | | wayne10