Dates in URL's
-
I have an issue of duplicate content errors and duplicate page titles which is penalising my site. This has arisen because a number of URLs are suffixed by date(s) and have been spidered . In principle I do not want any url with a suffixed date to be spidered.
Eg:-
www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm/06_07_13/13_07_13
http://www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm/20_07_13/27_07_13
Only this URL should be spidered:-
http://www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm
I have over 10,000 of these duplicates and firstly wish to remove them on block from Google ( not one by one ) and secondly wish to amend my robots.txt file so the URL's are not spidered. I do not know the format for either.
Can anyone help please.
-
Thanks Kyle.
Particularly grateful for the Disallow format, they are the only URL's using an underscore so will work for me. WIll be checking why these are being created.
Do I need to remove them using the Removal Tool in Google, is there a format for doing this on block ?
Thanks again,
Alan
-
Hi Alan,
I would probably start by adding a disallow rule to robots.txt.
**Disallow: /*_** _may work and block all your dated URLs from being indexed but may also have adverse affects if you have any URLs containing underscores. To test whether this solution would work I would firstly implement a disallow directly on a chosen dated URL, _**Disallow: /20_07_13 **_for example, and then test whether Google has noindexed the page. GWT should tell you whether you have inadvertently blocked any other pages by doing so.
You should also be thinking about how these URLs are being created and taking actions to prevent it. Consider implementing canonical tags if you haven't already to clean up any potential duplication issues.
Cheers,
K
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Value of URL Changes
Hi Guys, I have a question. Each product listed on my webstie has product number like /product.php?id=3624. After I spent many hours with MOZ, I figured out that this approach is wrong and I should use the product name as URL to achieve better SEO performance. Now I am planing to change the URL generating algoritm but should I do it for existing products. Some of them have already been linked to external websites. I am thinking to create mirror URLs but this may cause rather damage on my website. Do you know what is the right answer? Best, Tony
On-Page Optimization | | Threeding.com0 -
With 301 Redirects Does Changing URLs Matter?
We are redesigning our website in order to give it a more modern visual look. For the most part all the content will remain the same. Our old site is hosted on .asp so all of our current URLs look something like this: www.example.com/products/food.asp We plan on using 301 redirects in order to update every URL and remove the .asp. Since we are going to be doing 301 redirects for every existing URL anyways, does it matter from an SEO and ranking standpoint, if we also change the content and structure of the URL? For example, would we see a ranking impact if we were to change the above example URL to www.example.com/food? Obviously we want to try to retain as much link juice and ranking factors as possible during this redesign. Another issue we are seeing is with the image file names of our existing website images. We are moving to a new CMS platform (WordPress) that automatically saves images using a folder path similar to this: wp-uploads/2015-08/food. Will that change affect our SEO or ranking at all? When Google crawls an image does it care about the full path? Any insight would be much appreciated! 🙂
On-Page Optimization | | BlueLinkERP0 -
Changing existing URL's to improve SEO
Here's a general question: At what PR/page rank (or Moz 'page authority') would you no longer change an existing URL that's cryptic to one that includes the related page keyword or at least relevant terms? Does using a 301 redirect to the new URL preserve the page rank? Thanks!
On-Page Optimization | | flyntime_tx0 -
Competitor's 'hidden' links harming my site?
Hi everyone, I'm new to both Moz & seo, and am attempting to tackle our site's issues after being hit by panda / penguin, so would be grateful for any advice offered. I bought a website 3 years ago after the previous company that ran it went into administration. Having bought the website, it became apparent that the employees of the previous company had copied the entire site content, and relaunched it with a new look / brand. Over the last 3 years they've rewritten much of the content, but there remains a lot of links from their site back to ours which have had the anchor text stripped out, and point to images on our site which have since been removed, example below... <a href="http://www.MyCompany.com/catalog/images/filename.pdf" target="<a class="attribute-value">_blank</a>"><strong>strong>a> What I'm trying to understand is whether the 404 errors being returned by the broken links, and the presence of 'hidden' links on their site, is likely to reflect badly on our site or theirs? I'm not interested in outing anyone here, and I realise the standard recommendation for these kinds of situations is to write to the company telling them to remove the offending content, but if at all possible I'd prefer to fix our site by improving content & links etc, rather than 'force' them to take action and inadvertently improve their own site's content / rankings. As I say, all advice gratefully received 🙂
On-Page Optimization | | Sandy_M0 -
URL best practices
Hi, I have a problem here, I used http://www.vietnamvisacorp.com/faqs.html instead of http://www.vietnamvisacorp.com/faqs. Hence, http://www.vietnamvisacorp.com/faqs will be caused 404 page. My question is should I change from faqs.html to faqs (no .html)? Thanks in advance any advice?
On-Page Optimization | | JohnHuynh0 -
How do I get this program to see url with www. and with out www the same
The program is showing pages with www. as a differant page from a page with out the www. first, this is showing up as duplicate pages when they are the same page, how do I filter this?
On-Page Optimization | | masterplumbertom0 -
Which method should I use for my URL structure?
I have an existing site that is currently utilizing a structure that is like this: http://www.mysite.com/Ohio/City-of-Cleveland-PRODUCT-NAME Should I restructure it like: http://www.mysite.com/Ohio/City-of-Cleveland/Product-Name We are doing very well with very specific searches already but are sometimes coming in 2nd and 3rd place. For example: If I search for CLEVELAND PRODUCT NAME I always come up in the top three and about 60% of the time I am #1. I want to make it better. We have only launched in 4 states but plan on launching an additional 4 states over the next few weeks and I want to make sure we are building things properly. Any feedback would be wonderful. As usual, thanks everyone!! -Alex
On-Page Optimization | | dbuckles0 -
How woud you deal with Blog TAGS & CATEGORY listings that are marked a 'duplicate content' in SEOmoz campaign reports?
We're seeing "Duplicate Content" warnings / errors in some of our clients' sites for blog / event calendar tags and category listings. For example the link to http://www.aavawhistlerhotel.com/news/?category=1098 provides all event listings tagged to the category "Whistler Events". The Meta Title and Meta Description for the "Whistler Events" category is the same as another other category listing. We use Umbraco, a .NET CMS, and we're working on adding some custom programming within Umbraco to develop a unique Meta Title and Meta Description for each page using the tag and/or category and post date in each Meta field to make it more "unique". But my question is .... in the REAL WORLD will taking the time to create this programming really positively impact our overall site performance? I understand that while Google, BING, etc are constantly tweaking their algorithms as of now having duplicate content primarily means that this content won't get indexed and there won't be any really 'fatal' penalties for having this content on our site. If we don't find a way to generate unique Meta Titles and Meta Descriptions we could 'no-follow' these links (for tag and category pages) or just not use these within our blogs. I am confused about this. Any insight others have about this and recommendations on what action you would take is greatly appreciated.
On-Page Optimization | | RoyMcClean0