Dates in URL's
-
I have an issue of duplicate content errors and duplicate page titles which is penalising my site. This has arisen because a number of URLs are suffixed by date(s) and have been spidered . In principle I do not want any url with a suffixed date to be spidered.
Eg:-
www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm/06_07_13/13_07_13
http://www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm/20_07_13/27_07_13
Only this URL should be spidered:-
http://www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm
I have over 10,000 of these duplicates and firstly wish to remove them on block from Google ( not one by one ) and secondly wish to amend my robots.txt file so the URL's are not spidered. I do not know the format for either.
Can anyone help please.
-
Thanks Kyle.
Particularly grateful for the Disallow format, they are the only URL's using an underscore so will work for me. WIll be checking why these are being created.
Do I need to remove them using the Removal Tool in Google, is there a format for doing this on block ?
Thanks again,
Alan
-
Hi Alan,
I would probably start by adding a disallow rule to robots.txt.
**Disallow: /*_** _may work and block all your dated URLs from being indexed but may also have adverse affects if you have any URLs containing underscores. To test whether this solution would work I would firstly implement a disallow directly on a chosen dated URL, _**Disallow: /20_07_13 **_for example, and then test whether Google has noindexed the page. GWT should tell you whether you have inadvertently blocked any other pages by doing so.
You should also be thinking about how these URLs are being created and taking actions to prevent it. Consider implementing canonical tags if you haven't already to clean up any potential duplication issues.
Cheers,
K
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Url too long
Hi we have a wordpress website for Office Furniture Our domain name does not have office furniture in it. So we went from having domain.com/shop/category/second-sub-cat to domain.com/office-furniture/category/second-category However we now have 509 products flagged with too long urls I am wondering whether we should change back : For example domain.com/office-furniture/office-corner-desks/beech-corner-desks Should I go back to domain.com/shop/office-corner-desks/beech-corner-desks Or am I going to confuse matters even more? I should say this only affects categories as the products themselves are /shop/product I am just concerned that changing the re-direct in place originally will just make matters worse and confuse things - would I get more value changing back? Or should I stick with it and just try to shorten urls individually by product titles etc
On-Page Optimization | | KellyDSD860 -
Strange URL resulting a page
Hi, my friend has asked me to take a look at his site. I only know the basics of SEO so I'm learning along the way. He has some duplicate title errors showing in Moz, resulting to this page: https://www.domainname.com/about/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers This URL shows the 'About' page. I have tonnes of pages like this showing with really long URLs that result an actual page. Has anyone seen something like this before? I don't have a clue how this is showing the about page Any advice is greatly appreciated. Thanks James
On-Page Optimization | | Craze_Media0 -
Schema and Rich Snippets What's the difference?
Sorry if this is a daft question but... what is the difference between Rich snippets and Schema markup? Are they one and the same? They seem to be used interchaneably and I'm confused. If someone could give a brief sentence or two about the differences between them that would be great. Thanks
On-Page Optimization | | AL123al1 -
My company's product is referred to by two different names (SVN and Subversion). When cleaning up our Title tags, is it OK to use either name to keep the title tags around 70 characters?
I am cleaning up title tags that are too long or not correct. In our title tag we reference our product (a version of OSS source code). This product is often referred to as both SVN or Subversion. When writing Title tags is it OK to use one or the other depending on the length of the Title Tag? For instance: Contact Us | Free SVN & Git Hosting | Bug & Issue tracking | CloudForge vs **About CloudForge | Free Subversion & Git Hosting | Bug Tracking ** | |
On-Page Optimization | | CollabNet0 -
What's a reasonable bounce rate for school website?
Does anyone have a baseline on what the average bounce rate should be on a school website?
On-Page Optimization | | BillyBobGriffin0 -
Why there's a full-stop in the title of SEOMOZ's home page?
Hello, I see there's a full-stop (.) in the title of SEOMOZ's home page. Why is it so? Regards
On-Page Optimization | | IM_Learner0 -
What URL Should I use in Google Place Page?
Alright, I have a client that has 1 website and 14 locations. We want to create place pages for each of their locations but my question is which URL should I put in the place page and why? I can put in the root domain into each place page, or should I put in the URL that lands on the actual location on the root. example: domain.com/location1 Thanks!
On-Page Optimization | | tcseopro0 -
Using magentos own url re-writes
We are changing ecommerce platforms. Is it best to use magentos own url re-writes to redirect every page of a site from its old url to it new one?
On-Page Optimization | | LadyApollo0