What reasons exist to use noindex / robots.txt?
-
Hi everyone. I realise this may appear to be a bit of an obtuse question, but that's only because it is an obtuse question. What I'm after is a cataloguing of opinion - what reasons have SEOs had to implement noindex or add pages to their robots.txt on the sites they manage?
-
Many reasons. You don't want the admin pages of your site indexed, for example. You may not want all of the search queries that people perform on your site search to be indexed. You don't want or need your cart checkout being indexed for an ecommerce site. You don't want a print version and a web version of the same document indexed, so you exclude the print version from being indexed. Your site is in development, and you don't want it being indexed before it is ready.
For robots.txt in particular, some search engines now respect wildcards and you can exclude some session IDs via robots.txt. OSCommerce is real bad about creating session IDs and getting those indexed, then you have tons of different URLs indexed for the same page.
http://www.cogentos.com/bloggers-guide-to-using-robotstxt-and-robots-meta-tags-to-optimise-indexing/ is a post that explains some of the reasons to use robots and no-index on a Wordpress site.
-
There are a couple that come to my mind when i used them working for an agency. I remember one client had some temporary pages that didn't want to get indexed, explaining certain problem with a product at that time. We wanted the page to be live, but didn't want the problems that the product was having to show up in the search engines since it was just temporary.
Also, pages that are targeting same keywords that you dont want to erase or redirect and instead want to keep them live but at the same time you dont want to compete with the other main page. You just block it to the search engines.
Hope this helps
-
I really should have worded my question better. I'll try again.
**What reasons do people have for not wanting their pages show on search results? **
I've got a few reasons of my own, but I'm interested in seeing if there's any I hadn't thought of.
-
For pages you don't want them to show up on search results. =P
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Increasing in 404 errors that doesnt exist
Hi First of all I should say that I have error in the old webmaster not the new one. I have two WordPress blog in root and subfolder. Today I checked my webmaster and recognize that I had 100 errors (404) that found in few days ago. My root WordPress is OK but subfolder WordPress has error. Let me show you by example. http://example.com/subfolder/article15245 I had error for this page: http://example.com/article15245 looks like this subfolder deleted I checked my links, but all of them were OK and linked to the right URL. unfortunately this errors dont have "linked from" section
On-Page Optimization | | meysamzare7112365410 -
Header/Menu Links
Hey there, I seem to have heard mixed results on this. We have a LOT of pages that we are trying to tell google are important. Does it make that big of a difference if those pages are linked in the homepage/header menu? Or mega menu for example? We are essentially a review site, so we potentially have thousands of links that could be implemented into the menu. Thoughts are greatly appreciated!
On-Page Optimization | | HashtagHustler0 -
Using Bold text for keywords
Hello I am updating an old e-commerce website of mine and many keywords are in bold - shall I remove the bold tag or keep them there? This is for SEO.
On-Page Optimization | | xdunningx0 -
I have a site with jokes. What schema markup could I use?
My site is about jokes. I wonder what schema markup could I use to be more visible in the search results.
On-Page Optimization | | MichaelJanik0 -
Is there a tool that I can use to scrape and see metatags?
Looking for a tool that allows me to scrape the websites off a page listing of Google and output a spreadsheet with the websites and their related meta-tag details (mainly title tag). Is there a tool out there that can conveniently allow me to do this?
On-Page Optimization | | Gavo0 -
How to use canonical with mobile site to main site
I am pretty sure that the mobile version of the main site needs to be the same canonical link from what I understand. I am trying to find good docuementation that supports this. Even better if its from Google or Matt Cutts. I have a main domain like http://www.mydomain.com the mobile version of this is http://www.mydomain.com/m/ Should my canonical be rel="canonical" href="http://www.mydomain.com"/> for both these pages?
On-Page Optimization | | cbielich0 -
Will google see you bold/emphasis words if done in css?
We do not use header tags in our website. I understand bolding or emphsising words can be equally as effective but if done so in css will the google crawlers and spiders etc be able to put a weighted value on this style of code???
On-Page Optimization | | gsbureau0 -
Creating New Pages Versus Improving Existing Pages
What are some things to consider or things to evaluate when deciding whether you should focus resources on creating new pages (to cover more related topics) versus improving existing pages (adding more useful information, etc.)?
On-Page Optimization | | SparkplugDigital0