Dates in URL's
-
I have an issue of duplicate content errors and duplicate page titles which is penalising my site. This has arisen because a number of URLs are suffixed by date(s) and have been spidered . In principle I do not want any url with a suffixed date to be spidered.
Eg:-
www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm/06_07_13/13_07_13
http://www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm/20_07_13/27_07_13
Only this URL should be spidered:-
http://www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm
I have over 10,000 of these duplicates and firstly wish to remove them on block from Google ( not one by one ) and secondly wish to amend my robots.txt file so the URL's are not spidered. I do not know the format for either.
Can anyone help please.
-
Thanks Kyle.
Particularly grateful for the Disallow format, they are the only URL's using an underscore so will work for me. WIll be checking why these are being created.
Do I need to remove them using the Removal Tool in Google, is there a format for doing this on block ?
Thanks again,
Alan
-
Hi Alan,
I would probably start by adding a disallow rule to robots.txt.
**Disallow: /*_** _may work and block all your dated URLs from being indexed but may also have adverse affects if you have any URLs containing underscores. To test whether this solution would work I would firstly implement a disallow directly on a chosen dated URL, _**Disallow: /20_07_13 **_for example, and then test whether Google has noindexed the page. GWT should tell you whether you have inadvertently blocked any other pages by doing so.
You should also be thinking about how these URLs are being created and taking actions to prevent it. Consider implementing canonical tags if you haven't already to clean up any potential duplication issues.
Cheers,
K
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Url too long
Hi we have a wordpress website for Office Furniture Our domain name does not have office furniture in it. So we went from having domain.com/shop/category/second-sub-cat to domain.com/office-furniture/category/second-category However we now have 509 products flagged with too long urls I am wondering whether we should change back : For example domain.com/office-furniture/office-corner-desks/beech-corner-desks Should I go back to domain.com/shop/office-corner-desks/beech-corner-desks Or am I going to confuse matters even more? I should say this only affects categories as the products themselves are /shop/product I am just concerned that changing the re-direct in place originally will just make matters worse and confuse things - would I get more value changing back? Or should I stick with it and just try to shorten urls individually by product titles etc
On-Page Optimization | | KellyDSD860 -
To update or not to update news URLs ?
We manage a huge daily news website in my small country - keeping this a bit mysterious in case competitors are reading 🙂 Our URL structure is www.companyname.com/news/categoryofnews/title-of-article?id=articleid In this hyperreactive news world, title of articles change frequently (may be ten times a day for the main stories). The question we debate is : should we reflect the modification of the title in the URL or not ? Example : "Trump says he wants to ban search engines" would have URL http://www.companyname.com/news/entertainment/Trump-says-he-wants-to-ban-search-engines?id=12345678 Later in the day the title becomes "Trump denies he suggested banning search engines". Should the URL be modified to http://www.companyname.com/news/entertainment/Trump-denies-he-suggested-banning-search-engines?id=12345678 (option A) or not (option B) ? In Google News it makes no difference because of the sitemap, but in Google organic things are different. At present (option B in place), Google apparently doesn't see that the article has been updated, and shows the initial timestamp which is visually (and presumably SEOwise) not good : our new news looks like old news. Modifiying the URL would solve that issue, but could, may be, create another one : the new URL, being considered a new article, would lose, the acquired weight of the previous one in terms of referrals, social trafic and so on. Or not ? What do you think is the best option ? Thanks for your expertise, Yves
On-Page Optimization | | yves678901 -
My site's articles seem to never show up in Google.
This is in regards to a previous post that was answered for me:
On-Page Optimization | | Ctrl-Alt-Success
http://moz.com/community/q/my-site-s-name-not-ranking-in-google I was talking to a friend and he suggested I try to type in an article in google with the exact name followed by my site's domain name without the .com For example, I have an article entitled: "MULTITASKING IS BAD FOR YOU, MKAY?" Obviously it's a title most would not word in that way. I typed it in and followed it up with my site's domain minus .com. So "MULTITASKING IS BAD FOR YOU, MKAY? ctrl-alt-success" But I'm not even getting listed in the search. There's got to be something I'm missing. I understand backlinks are important for ranking, but when I'm trying to find an exact match along with my site's url minus the .com? I just have this strong hunch that something is awry. NOTE: It seems this is only with google. If I use Bing or Yahoo, it comes up just fine.0 -
Language for URLs on new international websites
We are due to launch our new site and it will be targeting an international market. We have setup these new sites in the following way www.website.com/fr/content-goes-here www.website.com/es/content-goes-here This has been done in conjunction with setting up the parameters in GWT and making sure it is optimised for the language itself, and that countries search engine. But our web dev team have said that the URLs at the moment will be in english and not the native language, so if you were on the french version of our site you would see the url in english and not french. Will this negatively affect the site for SEO, and who else would think it would be negative from a usability perspective? Any help is appreciated.
On-Page Optimization | | RyanCrawf19840 -
Content with changing URL and duplicate content
Hi everyone, I have a question regarding content (user reviews), that are changing URL all the time. We get a lot of reviews from users that have been dining at our partner restaurants, which get posted on our site under (new) “reviews”. My worry however is that the URL for these reviews is changing all the time. The reason for this is that they start on page 1, and then get pushed down to page 2, and so on when new reviews come in. http://www.r2n.dk/restaurant-anmeldelser I’m guessing that this could cause for serious indexing problems? I can see in google that some reviews are indexed multiple times with different URLs, and some are not indexed at all. We further more have the specific reviews under each restaurant profile. I’m not sure if this could be considered duplicate content? Maybe we should tell google not to index the “new reviews section” by using robots.txt. We don’t get much traffic on these URLs anyways, and all reviews are still under each restaurant-profile. Or maybe the canonical tag can be used? I look forward to your input. Cheers, Christian
On-Page Optimization | | Christian_T2 -
The value of changing URL structure
Dear Moz members, There have been many questions on this forum on this topic but I cannot find one that completely answers my question. We launched our new website about 7 months ago and the website contains around 3.000 product pages. The average page authority of the product pages is quite still quite low (12). The URL structure is built like this: www.website.com/catalog/ID/productname/ (with right keywords in the product name). So e.g the current URL is Our competitors rank higher on certain keywords while page authority (and DA) are significantly lower. Their URL structure is set up like this: www.website.com/productname/. Our most import keyword is "grafmonumenten" and the link we would like to rank on is: https://www.denhollandsche.nl/grafmonumenten/ My question now is:
On-Page Optimization | | stepsstones
1: how important is the length/depth of the URL structure?
2: is it beneficial for us to change the structure (www.website.com/productname/) use permanent redirects? My expierence is that changing the 'page url' can cause a short term drop in the serps, but can have positive effects on the longer term. Thanks for helping me out!0 -
301 redirects / clean urls
hello everyone, we moved our site to a new platform that has url rewrite feature. we are building out a spreadsheet of the old aspx urls and the corresponding page on the new site. my dev tells me that its not important to redirect the old page to the new "clean url" page which sounds a bit odd to me ..example below. okay. so he says its the same difference to 301 redirect this url: http://72.3.181.97/catalog/DesignerDogBeds.aspx to this urls: https://www.k9electronics.com/product.php?productid=3474 instead of this clean url: http://www.k9electronics.com/designer-dog-beds/ can anyone give me any feedback on this? thanks!
On-Page Optimization | | k9byron0 -
Rethinking company's monthly content production process.
I'm trying to rethink my company's content production process. I believe that we're stuck using a formula that works but can surely be improved. Our Current Process It essentially boils down to posting a certain number of content pieces per month for each client. After the pages are approved and live, there isn't much thought given to them. What We're Thinking After taking a step back, we realize now that a lot of these clients have sites with a tremendous amount of content that is rarely, if ever, revisited. In hopes of creating higher quality content and avoiding having to write that certain number of pieces per month, we're investigating alternative strategies to ensure each client has fresh content. What We're Looking Into Page Edits/Refreshes - I'm beginning to wonder if we can get similar gains by simply refreshing the content that already exists. We can include additional keywords and improve the content in a fraction of the time that it takes to produce a new piece. We're struggling to come up with a process for refreshing the content, however. Ideally we'd be implementing a process where content is revisited 6-12 months, but that still doesn't take care of the problem of creating too much new content. Simplified Version I believe that my company is creating too much content. Editing/refreshing seems like a better use of resources, but I have no idea how to implement a process and develop procedures. Questions What does your content production process look like? Do you produce a certain number a month, a quarter, as needed, etc? How do you go about refreshing your content?
On-Page Optimization | | SeoWebMechanix0