What reasons exist to use noindex / robots.txt?
-
Hi everyone. I realise this may appear to be a bit of an obtuse question, but that's only because it is an obtuse question. What I'm after is a cataloguing of opinion - what reasons have SEOs had to implement noindex or add pages to their robots.txt on the sites they manage?
-
Many reasons. You don't want the admin pages of your site indexed, for example. You may not want all of the search queries that people perform on your site search to be indexed. You don't want or need your cart checkout being indexed for an ecommerce site. You don't want a print version and a web version of the same document indexed, so you exclude the print version from being indexed. Your site is in development, and you don't want it being indexed before it is ready.
For robots.txt in particular, some search engines now respect wildcards and you can exclude some session IDs via robots.txt. OSCommerce is real bad about creating session IDs and getting those indexed, then you have tons of different URLs indexed for the same page.
http://www.cogentos.com/bloggers-guide-to-using-robotstxt-and-robots-meta-tags-to-optimise-indexing/ is a post that explains some of the reasons to use robots and no-index on a Wordpress site.
-
There are a couple that come to my mind when i used them working for an agency. I remember one client had some temporary pages that didn't want to get indexed, explaining certain problem with a product at that time. We wanted the page to be live, but didn't want the problems that the product was having to show up in the search engines since it was just temporary.
Also, pages that are targeting same keywords that you dont want to erase or redirect and instead want to keep them live but at the same time you dont want to compete with the other main page. You just block it to the search engines.
Hope this helps
-
I really should have worded my question better.
I'll try again.
**What reasons do people have for not wanting their pages show on search results? **
I've got a few reasons of my own, but I'm interested in seeing if there's any I hadn't thought of.
-
For pages you don't want them to show up on search results. =P
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does DA/PA have any effect on rankings?
I have seen many people are concerned about increasing DA and PA of their websites. While I am very curious why do people focus on increasing DA and PA? Does DA and PA effect the rankings of the website? Because I have recently launched my website regarding men beard trimmer and it is ranking on 1st page but not on number 1 position. Will increasing DA/PA of the site help me in occupying 1st position?
On-Page Optimization | | RyanAmin0 -
Two Robots.txt files
Hi there Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one. Thanks in advance! Cheers,
On-Page Optimization | | TrulyTravel
Satla0 -
Duplicate Title Tags/Meta Tags for Website with Multiple Locations
I currently have an insurance website that has over 40 offices in Ontario. The site also provides online quoting.
On-Page Optimization | | MainstreamMktg
In some of our programming, we have implemented variables in the URLS that will allow the quotes to go to the specific city offices - the issue I am having is that the quote in itself is the same quote form (same title, same meta) because it's technically one page on the website. We did it this way to avoid having to update 40 forms if a field on the form were to change. Is there any way I can relieve my site of this duplicate title tag/meta tag issue? Any insight would be really appreciated - thanks so much!0 -
Recommended Schema for a Collection/Category page?
Hi There! Taking on a small project up updating and adding in Schema to a clients site; a previous developer half put in data vocabulary. In my planning I was wondering if their would be a best schema type for category page of products - or a collection of products? Any ideas and experience? Thanks!
On-Page Optimization | | paul-bold0 -
Internal anchor text. Should we still use keywords?
I have a review site that has approximately 70 category pages. I'd like to include a few of them (not all 70) in the footer of the page for easy navigation and to direct the crawlers to the best ones. Is it advisable to use anchor text of "Category + reviews" or just leave it as "Category". I certainly dont want it to be overoptimized, but I do think its a good usability cue, so I'd be surprised if it was against guidelines. Any thoughts?
On-Page Optimization | | jim_shook0 -
Google Index/Cashe questions
I have 15k+ pages. I have 4.5k pages indexed. What relation is the google cashe to indexing pages? My site gets cashed every two days. The competition in my SERP goes 2-3weeks to get cashed. What does this indicate? Is your cashe date your last google crawl? How can I get google to crawl my site? Is there a way I can get google to crawl my site starting from an internal page. This way I could set up a better linking structure that would benefit from doing activities that get that page indexed to help get my site indexed more thoroughly...
On-Page Optimization | | JML11790 -
Software/tool for Content Inventory
Is there any tool out there which can crawl a section of a website (or whole website) and gives a list of urls by subsections? I have tried PowerMapper but it only gives visual map, not a list. Thanks!
On-Page Optimization | | StickyRiceSEO0 -
Should I use a Page Name variable after the ? for a dynamic web page
I'm converting for static to dynamic web pages. It appears that the page name is used for page ranking in the search engines. Will adding a Page Name variable help to increase our SEO. For example aspecialgift.com/subcat.php?PageName=GiftPage&ProductID=ABCDE. Does the page name variable make a difference?
On-Page Optimization | | NCBob0