What reasons exist to use noindex / robots.txt?
-
Hi everyone. I realise this may appear to be a bit of an obtuse question, but that's only because it is an obtuse question. What I'm after is a cataloguing of opinion - what reasons have SEOs had to implement noindex or add pages to their robots.txt on the sites they manage?
-
Many reasons. You don't want the admin pages of your site indexed, for example. You may not want all of the search queries that people perform on your site search to be indexed. You don't want or need your cart checkout being indexed for an ecommerce site. You don't want a print version and a web version of the same document indexed, so you exclude the print version from being indexed. Your site is in development, and you don't want it being indexed before it is ready.
For robots.txt in particular, some search engines now respect wildcards and you can exclude some session IDs via robots.txt. OSCommerce is real bad about creating session IDs and getting those indexed, then you have tons of different URLs indexed for the same page.
http://www.cogentos.com/bloggers-guide-to-using-robotstxt-and-robots-meta-tags-to-optimise-indexing/ is a post that explains some of the reasons to use robots and no-index on a Wordpress site.
-
There are a couple that come to my mind when i used them working for an agency. I remember one client had some temporary pages that didn't want to get indexed, explaining certain problem with a product at that time. We wanted the page to be live, but didn't want the problems that the product was having to show up in the search engines since it was just temporary.
Also, pages that are targeting same keywords that you dont want to erase or redirect and instead want to keep them live but at the same time you dont want to compete with the other main page. You just block it to the search engines.
Hope this helps
-
I really should have worded my question better.
I'll try again.
**What reasons do people have for not wanting their pages show on search results? **
I've got a few reasons of my own, but I'm interested in seeing if there's any I hadn't thought of.
-
For pages you don't want them to show up on search results. =P
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google ranking content for phrases that don't exist on-page
I am experiencing an issue with negative keywords, but the “negative” keyword in question isn’t truly negative and is required within the content – the problem is that Google is ranking pages for inaccurate phrases that don’t exist on the page. To explain, this product page (as one of many examples) - https://www.scamblermusic.com/albums/royalty-free-rock-music/ - is optimised for “Royalty free rock music” and it gets a Moz grade of 100. “Royalty free” is the most accurate description of the music (I optimised for “royalty free” instead of “royalty-free” (including a hyphen) because of improved search volume), and there is just one reference to the term “copyrighted” towards the foot of the page – this term is relevant because I need to make the point that the music is licensed, not sold, and the licensee pays for the right to use the music but does not own it (as it remains copyrighted). It turns out however that I appear to need to treat “copyrighted” almost as a negative term because Google isn’t accurately ranking the content. Despite excellent optimisation for “Royalty free rock music” and only one single reference of “copyrighted” within the copy, I am seeing this page (and other album genres) wrongly rank for the following search terms: “free rock music”
On-Page Optimization | | JCN-SBWD
“Copyright free rock music"
“Uncopyrighted rock music”
“Non copyrighted rock music” I understand that pages might rank for “free rock music” because it is part of the “Royalty free rock music” optimisation, what I can’t get my head around is why the page (and similar product pages) are ranking for “Copyright free”, “Uncopyrighted music” and “Non copyrighted music”. “Uncopyrighted” and “Non copyrighted” don’t exist anywhere within the copy or source code – why would Google consider it helpful to rank a page for a search term that doesn’t exist as a complete phrase within the content? By the same logic the page should also wrongly rank for “Skylark rock music” or “Pretzel rock music” as the words “Skylark” and “Pretzel” also feature just once within the content and therefore should generate completely inaccurate results too. To me this demonstrates just how poor Google is when it comes to understanding relevant content and optimization - it's taking part of an optimized term and combining it with just one other single-use word and then inappropriately ranking the page for that completely made up phrase. It’s one thing to misinterpret one reference of the term “copyrighted” and something else entirely to rank a page for completely made up terms such as “Uncopyrighted” and “Non copyrighted”. It almost makes me think that I’ve got a better chance of accurately ranking content if I buy a goat, shove a cigar up its backside, and sacrifice it in the name of the great god Google! Any advice (about wrongly attributed negative keywords, not goat sacrifice ) would be most welcome.0 -
Should I noindex our WordPress Categories?
What are the conditions under which we should noindex blog categories? I'm asking because I want to decided whether or not to noindex our categories. here.
On-Page Optimization | | Heydarian1 -
Does Google use 302's to pass value to the target page?
Hi, I've received the below advice, is this correct? Throughout the site, the 302 (moved temporarily) status code is used for redirects, which Google will use to pass value to the target page. Is this correct? I was under the impression a 301 was used to pass value to the target page? Could someone explain the difference between a 301 and a 302, I'm not 100% sure. Thanks, Nathan
On-Page Optimization | | Heehaw0 -
Integrate a blog within an existing site
Hello everyone! I work on a website of a very small company and so far no one has ever implemented (not even thought about) a proper content strategy. The only content on the site are products description. Through Analytics I discovered lots of opportunities and topics to be covered which would massively increase the traffic, increase the engagement and (hopefully) sales. Problem is that I don't really know how to integrate a blog into the existing site; my first thought was wordpress What is the best way to do it?
On-Page Optimization | | PremioOscar0 -
Should i avoid using H2 to optimize my home page
Hi, my site is www.in2town.co.uk and i am looking at ranking for lifestyle magazine but i am a bit worried. The titles of the articles on the home page are h2, and i am just wondering if i should change this to h3 as it may get confusing for google. Can anyone let me know if i should keep the article titles as h2 or if i am right that they need to be changed to h3 or something else any help would be great
On-Page Optimization | | ClaireH-1848860 -
What software do you use/work within for SEO?
Hi, Our site was put together in Dreamweaver and I'm not great at using html so I use Contribute to modify the info on our site. There are some limitations using Contribute so I'm wondering what other people use. Thanks for your input!
On-Page Optimization | | karlseidel0 -
Will a on-page food/drink menu take away from my target keywords?
I am trying to rank for Chicago corporate events, so I am writing new content for a landing page. I have a description of our venue and why one should have a corporate event there, all content that is very rich in corporate event keywords. I then go on to say our drink packages and food packages. Overall, my actual keyword targeted content is 1/5 of the content and the other 4/5 is menu. At first I thought the more content the better, but now that I realize a lot fo the content is just saying different foods and alcohols I'm questioning my strategy. I'm going to do my metas and h1s and stuff with corporate event keywords. But is the menu going to get in the way of ranking for the corporate event keywords? Thanks in advance 🙂
On-Page Optimization | | howlusa0 -
Using H3-4 tags in the footer or sidebars: good or not?
Howdy SEOmoz fans! Is it considered a good / bad / neutral practice to include H tags in the footer, as a mean to group a few links? Take http://www.seomoz.org/ for instance: - Voted Best SEO Tool 2010! = H2
On-Page Optimization | | AxialDev
- Looking for SEO consulting? = H3
- Product and Tools = H3 Company = H3 etc. I often see the same principle applied to sidebars. I feel like because they don't contribute to the actual content structure and because they are repeated from page to page, we should avoid them, but I have nothing to back my intuition. [+] Perhaps they are helpful for usability (screen readers) and thin added value (i.e. category names that carry more weight than if they weren't headers). What do you think? Thanks for your time.1