What reasons exist to use noindex / robots.txt?
-
Hi everyone. I realise this may appear to be a bit of an obtuse question, but that's only because it is an obtuse question. What I'm after is a cataloguing of opinion - what reasons have SEOs had to implement noindex or add pages to their robots.txt on the sites they manage?
-
Many reasons. You don't want the admin pages of your site indexed, for example. You may not want all of the search queries that people perform on your site search to be indexed. You don't want or need your cart checkout being indexed for an ecommerce site. You don't want a print version and a web version of the same document indexed, so you exclude the print version from being indexed. Your site is in development, and you don't want it being indexed before it is ready.
For robots.txt in particular, some search engines now respect wildcards and you can exclude some session IDs via robots.txt. OSCommerce is real bad about creating session IDs and getting those indexed, then you have tons of different URLs indexed for the same page.
http://www.cogentos.com/bloggers-guide-to-using-robotstxt-and-robots-meta-tags-to-optimise-indexing/ is a post that explains some of the reasons to use robots and no-index on a Wordpress site.
-
There are a couple that come to my mind when i used them working for an agency. I remember one client had some temporary pages that didn't want to get indexed, explaining certain problem with a product at that time. We wanted the page to be live, but didn't want the problems that the product was having to show up in the search engines since it was just temporary.
Also, pages that are targeting same keywords that you dont want to erase or redirect and instead want to keep them live but at the same time you dont want to compete with the other main page. You just block it to the search engines.
Hope this helps
-
I really should have worded my question better. I'll try again.
**What reasons do people have for not wanting their pages show on search results? **
I've got a few reasons of my own, but I'm interested in seeing if there's any I hadn't thought of.
-
For pages you don't want them to show up on search results. =P
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does DA/PA have any effect on rankings?
I have seen many people are concerned about increasing DA and PA of their websites. While I am very curious why do people focus on increasing DA and PA? Does DA and PA effect the rankings of the website? Because I have recently launched my website regarding men beard trimmer and it is ranking on 1st page but not on number 1 position. Will increasing DA/PA of the site help me in occupying 1st position?
On-Page Optimization | | RyanAmin0 -
Https://www.fitness-china.com/pilates-equipment How to find the most relevant internal link pages
https://www.fitness-china.com/pilates-equipment How to find the most relevant internal link pages? site this page, find https://www.fitness-china.com/pilates-equipment-kr and https://www.fitness-china.com/pilates-equipment-jp Is there any other better way
On-Page Optimization | | ahislop5740 -
Are there detrimental effects of having multiple robot tags
Hi All, I came across some pages on our site that have multiple robot tags, but they have the same directives. Two are identical while one is for Google only. I know there aren't any real benefits from having it set up this way, but are there any detrimental effects such as slowing down the bots crawling these pages? name="googlebot" content="index, follow, noodp"/> Thanks!
On-Page Optimization | | STP_SEO0 -
Stong Tags still useful in 2016
Hello, Are strong tags still beneficial in 2016? A few co-workers feel they look spammy on our site and that they are no longer really used by the search engines. However, we can't find any real significant info for no longer implementing them or saying they're outdated. Thanks!
On-Page Optimization | | STP_SEO0 -
Silly question about noindex and canonical
Hi, This is probably going to sound a bit stupid, but I nevertheless want to check. We have a site that's going to have identical pages (really not my choice) for a sales reason. The two examples would be example.com/profile-name and example.com/location/profile-name Users using the onsite navigation will always end up in the latter example naturally as they have to select a location before viewing content (plus having the location in the url is nice as there are multiple profiles across different locations that have the same name). However, it's easier to sell our services when we can offer just example.com/profile-name to users for their own marketing reasons. I'd like to make the example.com/profile-name noindex follow, and have just the example.com/location/profile-name indexed, but not sure if it would be better to implement canonical tags instead? Can anyone see any potential pitfalls of using either method or does it not really make a difference (which is what I suspect, but I'd rather look stupid than get this wrong)? Thanks!
On-Page Optimization | | LeahHutcheon0 -
Link juice/PR to social media platforms
Hi, Do you know if Google gives as much link juice/PR to social media platforms (when linking to them from a website), as the give to perhaps another internal/external link? The thought is: As Google is getting more intelligent and gives social media that much value in SERP, would the pass on/through link juice to the social media. Hope my question is understandable. Thks.
On-Page Optimization | | teambenns_seo0 -
My report is showing duplicate titles.. for http://www.mysite.com/ & http://mysite.com
Is this because I setup my campaign wrong in seo moz or should I have my site setup differently to forward all traffic without the www to the www version? Or vise versa? Thanks for any advice!
On-Page Optimization | | Fridaythe15th0 -
Can we listed URL on Website sitemap page which are blocked by Robots.txt
Hi, I need your help here. I have a website, and few pages are created for country specific. (www.example.com/uk). I have blocked many country specific pages from Robots.txt file. It is advisable to listed those urls (blocked by robots.txt) on my website sitemap. (html sitemap page) I really appreciate your help. Thanks, Nilay
On-Page Optimization | | Internet-Marketing-Profs0