I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?
-
Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?
-
When you say "people," are you saying your own web team duplicates content to make their job easier? Or am I missing something?...
If that's the case, you really should create unique URL's with unique page titles, product info, etc. That's the correct way to avoid getting hit for duplicate content - don't create it. It seems like what you're doing now is more of a band-aid solution to the problem.
I'd consider that even though creating unique content in situations like this can seem daunting and/or be more expensive, there's probably huge long-term gains to made if you do it right.
-
It is not bad, just not best practices because Google will still index the URL's if they are mentioned on other pages. Just to quote them:
"While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information..."
What I would do instead is either use rel="canonical" or 301 redirects. I hope that helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Keyword ranking for different page than the page optimized
I have optimized "equipment trailer for rent" on this page: http://www.bigtrailerrentals.com/flatbed-trailer-rentals/equipment-deckover. I'm wondering if anyone can tell me why Google has chosen to rank the keyword phrase for this page: http://www.bigtrailerrentals.com/flatbed-trailer-rentals/equipment-24 This is just one example. It has happened on several of my pages / keywords.
On-Page Optimization | | BigTrailerRentals0 -
Unique Pages with Thin Content vs. One Page with Lots of Content
Is there anyone who can give me a definitive answer on which of the following situations is preferable from an SEO standpoint for the services section of a website? 1. Many unique and targeted service pages with the primary keyword in the URL, Title tag and H1 - but with the tradeoff of having thin content on the page (i.e. 100 words of content or less). 2. One large service page listing all services in the content. Primary keyword for URL, title tag and H1 would be something like "(company name) services" and each service would be in the H2 title. In this case, there is lots of content on the page. Yes, the ideal situation would be to beef up content for each unique pages, but we have found that this isn't always an option based on the amount of time a client has dedicated to a project.
On-Page Optimization | | RCDesign741 -
Is the HTML content inside an image slideshow of a website crawled by Google?
I am building a website for a client and i am in a dilemma whether to go for an image slideshow with HTML content on the slides or go for a static full size image on the homepage. My concern is that HTML content on the slideshow may not get crawled by Google and hence may not be SEO friendly.
On-Page Optimization | | aravinn0 -
"irrelevant pages of a site"
Hi there! Some pages of my site like "contact" or "registration": Should they have a title and a description tag? They are pages that I don't want them to be shown in the SERPs....Could I be penalized by google If I don't do so? The SEOMOZ crawling tool warms me about this issue (to short titles, no meta-description tags....) Many thanks
On-Page Optimization | | juanmiguelcr0 -
Why is this site 1st in Google??
Hello The site www.woodensigns.net is 1st in google for the keyword "wooden Signs". All seo indicators are poor except keyword in url; wich, i thought, was not a + for google anymore. Could someone help me to understand here? Thank you Emmanuel
On-Page Optimization | | manu450 -
Pages that show in Open Site Explorer that show as 404's in search
When I use Site Explorer to find our Top Pages there are a whole series of pages like this: http://www.nile-cruises-4u.co.uk/mybestbets/mybestbets-culture.html which I think were created when we had a Bulletin Board on the site. Although they are showing in the Top Pages results when you visit the page you get the following error message: HTTP Error 404.0 - Not Found The resource you are looking for has been removed, had its name changed, or is temporarily unavailable. What implications result from these pages still showing in results and what action should we take without messing up our search rankings? Any advice would be gratefully received.
On-Page Optimization | | NileCruises0 -
My website is saying I have duplicate page content and page title. How do I fix it?
Hi, I created a website on webstarts.com. After I launched it then ran a scan through SEO it says I have duplicate page content and page title. The 2 pages it is reading are technically the same page. www.mobilemowermedicsinc.com and www.mobilemowermedicsinc.com/index . I am unsure how to get rid of on of these as it keeps saying this is an error in the SEO scan. Could someone please advise me of what to do from here. Thanks!
On-Page Optimization | | bcarp880 -
Old pages
I have a site where I have 5,000 new products each year, I never waned to deleted the old pages due to links pointing to them and keywords. But I now have 20,000 plus pages, does having that many pages spread out my link juice or does it effect me in any other ways over having a site with 5,000 pages or should I keep not deleting old pages so I dont loose any links? Along with that I currently do not link to my old pages from my site so Im guessing google does not get to them very often if at all, if you agree to still keep them should I link to them somewhere? Because the products are not that simiiar and they do bring added value I dont think canonical would work here
On-Page Optimization | | Dirty0