Dates in URL's
-
I have an issue of duplicate content errors and duplicate page titles which is penalising my site. This has arisen because a number of URLs are suffixed by date(s) and have been spidered . In principle I do not want any url with a suffixed date to be spidered.
Eg:-
www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm/06_07_13/13_07_13
http://www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm/20_07_13/27_07_13
Only this URL should be spidered:-
http://www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm
I have over 10,000 of these duplicates and firstly wish to remove them on block from Google ( not one by one ) and secondly wish to amend my robots.txt file so the URL's are not spidered. I do not know the format for either.
Can anyone help please.
-
Thanks Kyle.
Particularly grateful for the Disallow format, they are the only URL's using an underscore so will work for me. WIll be checking why these are being created.
Do I need to remove them using the Removal Tool in Google, is there a format for doing this on block ?
Thanks again,
Alan
-
Hi Alan,
I would probably start by adding a disallow rule to robots.txt.
**Disallow: /*_** _may work and block all your dated URLs from being indexed but may also have adverse affects if you have any URLs containing underscores. To test whether this solution would work I would firstly implement a disallow directly on a chosen dated URL, _**Disallow: /20_07_13 **_for example, and then test whether Google has noindexed the page. GWT should tell you whether you have inadvertently blocked any other pages by doing so.
You should also be thinking about how these URLs are being created and taking actions to prevent it. Consider implementing canonical tags if you haven't already to clean up any potential duplication issues.
Cheers,
K
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301 and the base URL
Hi, please bear with me as I'm pretty new to all this! I've my base URL but obviously want to add keywords to it for seo purposes. Should I redirect from the base URL to the URL with the keywords appended? So my landing page goes from say www.moz.com to www.moz.com/keywords-here. If I do that, should I replicate all the meta data (descriptions etc) on the original landing page? Or does it not matter? Thanks, Nick
On-Page Optimization | | nickwoodward0 -
Review rich snippet with reviews hosted on customer feedback company's website?
Is it possible to enable rich snippets for reviews that are not visible on the page in question? A restaurant's Wordpress website mentions the average rating but the actual customer feedback and reviews are displayed on a page on the website of the feedback company. Would it be enough to implement a small widget (displays star rating, number of reviews and links to the review page) and add microdata/J-SON to convince search engines that the reviews are legit? Thanks in advance! Bas
On-Page Optimization | | SEO-Bas0 -
Using keywords in my URL: Doing a redirect to /keyword
My website in "On Page Grade" received an A.Anyway, I only have 1 thing to optimize:_"Use Keywords in your URL__Using your targeted keywords in the URL string adds relevancy to your page for search engine rankings, assists potential visitors identify the topic of your page from the URL, and provides SEO value when used as the anchor text of referring links."_My website is ranking in top10 for a super high competitive keyword and all my others competitors have the keyword on their domain, but not for my URL.Since I can't change my domain for fixing this suggestion, I would like to know what do you think about doing a 301 redirect from / to mydomainname.com/keyword/So the index of my website would be the /keyword.I don't know if this can make a damage to my SERP for the big change ir it would be a great choice.
On-Page Optimization | | estebanseo0 -
Tips on URL structure for a site re-design
Wanted to know what you would do with regards to urls – in an ideal world how would you structure them? Keen to know as me and dave are soon to have a meeting about this and were wondering about changing them from the current – http://www.looking4parking.com/airport/gatwick to something like - www.looking4parking.com/gatwick-airport-parking We will soon have pages for the specific parking types that will be a lot more engaging to users with some really useful content on benefits, features, how a certain type of parking works, images, video etc. Currently going to a type of parking, such as meet and greet just brings up a dropdown modal – I was thinking of having the url structure looking like this – www.looking4parking.com/gatwick-airport-meet-and-greet-parking www.looking4parking.com/gatwick-airport-on-site-parking www.looking4parking.com/gatwick-airport-park-and-ride We will then have specific pages for each parking product – in which this product will have unique content built around it – each will have an overview of the product, benefits, features, reviews, images, directions to the car park, find your route and eventually a video on each product So for example we currently have the product “Jet Parks 2” at Manchester airport – the current url is - http://www.looking4parking.com/airport/manchester/park-and-ride/jetparks-2 I would like to change this now we have the opportunity to refresh the whole system, to something along the lines of **domain/location/product title - **www.looking4parking.com/manchester-airport-parking/jetparks-2 or as we have some similar products at certain airports (mainly where the airport has multiple terminals) we would just change it to the following - www.looking4parking.com/manchester-airport-parking/jetparks-3 What are peoples thoughts/opinions on the above?
On-Page Optimization | | RyanCrawf19840 -
Website redesign and it's impact on ranking
Hey Everyone, I have had a website in place for over 7 years and I am now at the point where I need to implement a redesign in order to sell our product more effectively. -URL's will stay the same
On-Page Optimization | | Justin45
-Content will change but be very similar page-to-page
-Title & Meta tags will remain the same I'm planning on taking the site from a non-backend site to a wordpress site so the navigation will change. Does anyone foresee that this change will have any dramatic effect on site ranking? Thank You!0 -
URL for location pages
Hello all We would like to create clean, easy URLs for our large list of Location pages. If there are a few URLs for each of the pages, am I right when I'm saying we would like this to be the canonical? Right now we would like the URL to be: For example
On-Page Optimization | | Ferguson
Domain.com/locations/Columbus I have found some instances where there might be 2,3 or more locations in the same city,zip. My conclusion for these would be: adding their Branch id's on to the URL
Domain.com/locations/Columbus/0304 Is this an okay approach? We are unsure if the URL should have city,State,zip for SEO purposes?
The pages will have all of this info in it's content
BUT what would be best for SEO and ranking for a given location? Thank you for any info!0 -
Rename index.php or keyword in URL?
It is important for me to get good search results for keyword + city name For instance: tulips amsterdam What would be better: renaming index.php or adding the cityname to the URL? www.example.com/amsterdam/tulips OR www.example.com/pages/tulips-amsterdam
On-Page Optimization | | svdg0 -
URL with two forward slashes //
We have a potential client with a URL structure in this fashion: http://www.site-url.com//cpage/page.html pretty strange, right? my question is: How bad are the 2 forward slashes // for SEO? How bad is it to have that extra layer of /cpage in the URL? this doesn't appear to serve any other purpose than making the URL longer than necessary.
On-Page Optimization | | Motava0