Dates in URL's
-
I have an issue of duplicate content errors and duplicate page titles which is penalising my site. This has arisen because a number of URLs are suffixed by date(s) and have been spidered . In principle I do not want any url with a suffixed date to be spidered.
Eg:-
www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm/06_07_13/13_07_13
http://www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm/20_07_13/27_07_13
Only this URL should be spidered:-
http://www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm
I have over 10,000 of these duplicates and firstly wish to remove them on block from Google ( not one by one ) and secondly wish to amend my robots.txt file so the URL's are not spidered. I do not know the format for either.
Can anyone help please.
-
Thanks Kyle.
Particularly grateful for the Disallow format, they are the only URL's using an underscore so will work for me. WIll be checking why these are being created.
Do I need to remove them using the Removal Tool in Google, is there a format for doing this on block ?
Thanks again,
Alan
-
Hi Alan,
I would probably start by adding a disallow rule to robots.txt.
**Disallow: /*_** _may work and block all your dated URLs from being indexed but may also have adverse affects if you have any URLs containing underscores. To test whether this solution would work I would firstly implement a disallow directly on a chosen dated URL, _**Disallow: /20_07_13 **_for example, and then test whether Google has noindexed the page. GWT should tell you whether you have inadvertently blocked any other pages by doing so.
You should also be thinking about how these URLs are being created and taking actions to prevent it. Consider implementing canonical tags if you haven't already to clean up any potential duplication issues.
Cheers,
K
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Home page keyword in url
I have been looking into SEO for a few weeks now trying to perfect a homepage. Going through various sources on MOZ, and other examples out there on the internet, I keep seeing that you should have your keyword in the URL of the page. The homepage is the page most people want to rank the highest in google searches, however, you cannot put the keyword in the URL as most home page URLs are simply /. Should I actually make the home like this: www.example.com/key-word-example? I would imagine this would not be the normal for many users and would seem like it's not the home page.
On-Page Optimization | | Matthew_smart0 -
How to transfer old WP blog to new URL
I have a 9 year old WP website with a WP blog which is still getting 300+ new visitors a day even though I have not written a blog for 5 years and have not updated content. Some posts have over 25,000 links. However the Moz analytics is fraught with significant errors-404 redirects, page not found, dup content, no metatags, title too long etc. I was totally inexperienced 5 years ago and made many errors. However the basic content was sound and still is producing new visitors. I am starting a new ecommerce website using the same name but the URL and server will be different. I want to transfer my WP blog to the new site. I am concerned however that bringing the posts over can create the same errors on the new site. If I update all of the blogs on the old site using Yoast before transferring the blog to the new site will that help. I suppose I could check those flagged dup content and only transfer one of that category?
On-Page Optimization | | wianno1680 -
Errors in URL´s
SEOMOZ is showing quite a lot of URL Errors like this: http://trampoliny.net.pl/akcesoria/pokrowiec-basic?frontend=1825cb1eea3af8ee6ee2d96617d32ff6 All these URL´s use the parameter "?frontend=". In webmaster tools we told google not to index this parameter. Unfortunately at the moment we cannot set this parameter as "NOINDEX". We also dont want to use a robots.txt file. How to get rid of the URLS in Seomoz?
On-Page Optimization | | drgoodcat0 -
Google's Page Layout Algorythm
It seems that Google have been or will penalizing websites with too many ads above the fold. Is it me or Google's search result layout is a perfect example of what NOT to do?
On-Page Optimization | | sbrault741 -
Different pages for OS's vs 1 Page with Dynamic Content (user agent), what's the right approach?
We are creating a new homepage and the product are at different stages of development for different OS's. The value prop/messaging/some target keywords will be different for the various OS's for that reason. Question is, for SEO reasons, is it better to separate them into different pages or use 1 page and flip different content in based on the user agent?
On-Page Optimization | | JoeLin0 -
Using magentos own url re-writes
We are changing ecommerce platforms. Is it best to use magentos own url re-writes to redirect every page of a site from its old url to it new one?
On-Page Optimization | | LadyApollo0 -
Excessive Internal Linking...But it's a product page. What to do?
A few of our companies sites' product pages have the warning about excessive internal links. But these pages are product pages (for example). Should we be worried about this warning? Are there ways to avoid it? Or is it just the nature of the beast...? Thanks in advance!
On-Page Optimization | | DevonIntl0 -
Duplicate content issue with dynamically generated url
Hi, For those who have followed my previous question, I have a similar one regarding dynamically generated urls. From this page http://www.selectcaribbean.com/listing.html the user can make a selection according to various criteria. 6 results are presented and then the user can go to the next page. I know I should probably rewrite url's such as these: http://www.selectcaribbean.com/listing.html?pageNo=1&selType=&selCity=&selPrice=&selBeds=&selTrad=&selMod=&selOcean= but since all the results presented are basically generated on the fly for the convenience of the user, I am afraid google my consider this as an attempt to generate more pages as there are pages for each individual listing. What is my solution for this? Nofollow these pages? Block them thru robots txt?
On-Page Optimization | | multilang0