Dates in URL's
-
I have an issue of duplicate content errors and duplicate page titles which is penalising my site. This has arisen because a number of URLs are suffixed by date(s) and have been spidered . In principle I do not want any url with a suffixed date to be spidered.
Eg:-
www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm/06_07_13/13_07_13
http://www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm/20_07_13/27_07_13
Only this URL should be spidered:-
http://www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm
I have over 10,000 of these duplicates and firstly wish to remove them on block from Google ( not one by one ) and secondly wish to amend my robots.txt file so the URL's are not spidered. I do not know the format for either.
Can anyone help please.
-
Thanks Kyle.
Particularly grateful for the Disallow format, they are the only URL's using an underscore so will work for me. WIll be checking why these are being created.
Do I need to remove them using the Removal Tool in Google, is there a format for doing this on block ?
Thanks again,
Alan
-
Hi Alan,
I would probably start by adding a disallow rule to robots.txt.
**Disallow: /*_** _may work and block all your dated URLs from being indexed but may also have adverse affects if you have any URLs containing underscores. To test whether this solution would work I would firstly implement a disallow directly on a chosen dated URL, _**Disallow: /20_07_13 **_for example, and then test whether Google has noindexed the page. GWT should tell you whether you have inadvertently blocked any other pages by doing so.
You should also be thinking about how these URLs are being created and taking actions to prevent it. Consider implementing canonical tags if you haven't already to clean up any potential duplication issues.
Cheers,
K
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does changing/shortening a url hurt SEO?
Hi all, I am in the process of making small optimization changes to my site. I noticed Moz identified quite a few URLs that could be shortened. I intend to shorten these URLs and create 301 redirects to ensure website users land on the right page. My question is, will this change in URL damage rankings and engagement(assuming the URL remains content relevant)? I have read in some places that when creating URL redirects for a change in domain, people saw a dip in rankings and engagement. I, however, am not intending to change the main domain of the site, but rather the URL slug. Any thoughts?
On-Page Optimization | | annegretwidmer0 -
Url shows up in "Inurl' but not when using time parameters
Hey everybody, I have been testing the Inurl: feature of Google to try and gauge how long ago Google indexed our page. SO, this brings my question. If we run inurl:https://mysite.com all of our domains show up. If we run inurl:https://mysite.com/specialpage the domain shows up as being indexed If I use the "&as_qdr=y15" string to the URL, https://mysite.com/specialpage does not show up. Does anybody have any experience with this? Also on the same note when I look at how many pages Google has indexed it is about half of the pages we see on our backend/sitemap. Any thoughts would be appreciated. TY!
On-Page Optimization | | HashtagHustler1 -
Ecommerce URLs with numbers
Hi everybody! I have to optimize an ecommerce where somebody has previously done the SEO optimization, although the URLs have numbers before the product's name They have told me that these numbers are useful to find the products, so I think it shouldn't be really bad if I don't redirect them to "clear" ones. For example: /colesterol-sobrepeso/2217-hc-grass-capsulas-duras-15-capsulas.html > /colesterol-sobrepeso/hc-grass-capsulas-duras-15-capsulas.html Am I right? After all, they contain the keywords and the subfolders are also ok. Or it would be better if I redirect the whole site? Thanks!
On-Page Optimization | | Estherpuntu0 -
Articles URL
Hello, Currently, I am parsing article base on article ID on URL request. For example:
On-Page Optimization | | JohnHuynh
To go to the content of article **What is the visa on arrival? How to get it? **I am using URL like that http://www.vietnamvisacorp.com/news/what-is-the-visa-on-arrival--how-to-get-it--245.html and base on ID 245 to get this page content.
But, now I want to optimize this URL to http://www.vietnamvisacorp.com/news/what-is-the-visa-on-arrival--how-to-get-it.html. Then I got a problem: How to get an article without ID? Thanks,0 -
Over 12.000 302's?
Hi. I'm monitoringssystem a magento webshop. It has more 12.000 temp 302 redirects. Is it also a problem if the redirects are for a nonimportant subpage, such as an enable-cookies page?
On-Page Optimization | | Budskab0 -
Blog URL
I know that this question has been asked in the past, and that website.com/blog is better for seo purposes than blog.website.com. We want to setup a custom blog on our site, using Wordpress. Our designers/host are telling us that buy using website.com/blog can causes issues b/c Wordpress is open source, and our site could be hacked? Is there anything we should do about this? Any suggestions? Any Advice appreciated!!! Thanks!
On-Page Optimization | | TP_Marketing0 -
URL Rewrite
(By Google Traductor) Hello, I wanted to ask about some changes that we are evaluating for the issue of passing the url with variables to be more descriptive, for example: http://www.agroads.com.ar/detalle.asp?clasi=139592 tohttp://www.agroads.com.ar/humedimetro-para-cereales-draminski-gmm-139592.html In this case corresponds to the breakdown of a product if you have long published andcan be well positioned to change the title of this position would be lost unless youmanage it with a 301, as one would manage when you have more than 30000 products and title may change several times? There are tools to manage this? Finally, we must apply this to all listed with their respective filters, recommends doingtheir part with 301 redirects and analyze what funciene well to continue with the rest or implement a complete change? I hope I can bring a little light to implement this. Greetings and thanks! Roberto
On-Page Optimization | | romaro0 -
Page URL Hiearchy
So I have read on here that page URL Hiearchy is important. My question is from a search engine standpoint which of the following methods would be the best to use (or another if not listed) COMPACT and naturally hierarchical MountainBiking.com MountainBiking.com/adventures ( a list of the pages below ) MountainBiking.com/adventures/in whistler (for each page) MountainBiking.com/adventures/in utah OR VERBOSE but reptetive MountainBiking.com MountainBiking.com/Mountain Biking adventures ( intro + a list of the pages below ) MountainBiking.com/Mountain Biking Adventures/Mounting Biking adventures in whistler MountainBiking.com/Mountain Biking Adventures/Mountain Biking Adventures in Utah It seemed like the blog I read suggested the compact form, but it seems to me that the verbose (though admittedly a bit clunky) seems better so far as exact keyword match etc. Experience and or advice on this?
On-Page Optimization | | bThere0