Dates in URL's
-
I have an issue of duplicate content errors and duplicate page titles which is penalising my site. This has arisen because a number of URLs are suffixed by date(s) and have been spidered . In principle I do not want any url with a suffixed date to be spidered.
Eg:-
www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm/06_07_13/13_07_13
http://www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm/20_07_13/27_07_13
Only this URL should be spidered:-
http://www.carbisbayholidays.co.uk/carbis-bay/houses-in-carbis-bay/seaspray.htm
I have over 10,000 of these duplicates and firstly wish to remove them on block from Google ( not one by one ) and secondly wish to amend my robots.txt file so the URL's are not spidered. I do not know the format for either.
Can anyone help please.
-
Thanks Kyle.
Particularly grateful for the Disallow format, they are the only URL's using an underscore so will work for me. WIll be checking why these are being created.
Do I need to remove them using the Removal Tool in Google, is there a format for doing this on block ?
Thanks again,
Alan
-
Hi Alan,
I would probably start by adding a disallow rule to robots.txt.
**Disallow: /*_** _may work and block all your dated URLs from being indexed but may also have adverse affects if you have any URLs containing underscores. To test whether this solution would work I would firstly implement a disallow directly on a chosen dated URL, _**Disallow: /20_07_13 **_for example, and then test whether Google has noindexed the page. GWT should tell you whether you have inadvertently blocked any other pages by doing so.
You should also be thinking about how these URLs are being created and taking actions to prevent it. Consider implementing canonical tags if you haven't already to clean up any potential duplication issues.
Cheers,
K
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why the url inspection is disabled in search console ?
In this situation, how can we make our pages be fetched by google?
On-Page Optimization | | supporthandle0 -
What word should I use in my URL for my blog
Should I use the word "blog" in my sub folder as in : http://www.mybusiness.com/blog or should I use http://www.mybusiness.com/news. Is there a difference for when my site is crawled. I understand that a blog works a little differently. Can someone explain the basics?
On-Page Optimization | | graemesanderson0 -
Changing a page url
I have a page that ranks well (#4) for a good keyword. However, the url has the keyword in it but is misspelled. I would like to change the url to have the correct spelling but do not want to lose the ranking that I have. What is the best and safest way to proceed?
On-Page Optimization | | bhsiao0 -
URL Question
This url looks bad: http://www.patrickmunoz.com/#!classes/c1vw1 And when you click around the page change doesn't actually occur, it's a fade into the next page. I think this is a major problem for rankings. Although pages are crawled: https://www.google.com/search?q=site%3Ahttp%3A%2F%2Fwww.patrickmunoz.com%2F&oq=site%3A&aqs=chrome.2.69i57j69i58j69i59l3j69i61.3548j0j7&sourceid=chrome&espv=210&es_sm=122&ie=UTF-8 When I search for a simple page - "patrick munoz FAQs" nothing comes up:
On-Page Optimization | | tylerfraser
https://www.google.com/search?q=site%3Ahttp%3A%2F%2Fwww.patrickmunoz.com%2F&oq=site%3A&aqs=chrome.2.69i57j69i58j69i59l3j69i61.3548j0j7&sourceid=chrome&espv=210&es_sm=122&ie=UTF-8#q=patrick+munoz+|+FAQs Do you think this is a bad url configuration? Thanks! Tyler0 -
Will Google Custom Search results on my home page kill it's ranking?
This is probably a dumb question, but here goes anyway. 🙂 On a site I have it would be very useful to the reader to offer a search box that uses a Google Custom Search that I have optimized to search websites that are closely on-topic with my site. I know it sounds bad that I would send people to other sites, but just assume that the reasons are valid for this discussion. My question is, if the search results are set to display on the same page (the home page) as the search box, will the links in the search results to external sites just bleed my page rank to death? I assume it would, but thought I'd check just in case I'm missing something. I have to option to place the results on separate page of my site, and noindex it, but it won't be as powerful as it would be on the home page.
On-Page Optimization | | bizzer0 -
Recommendation: Add a canonical URL tag referencing this URL to the header of the page.
Please clarify: In the page optimization tool, seomoz recommends using the canonical url tag on the unique page itself. Is it the same canonical url tag used when want juice to go to the original page? Although the canonical URL tag is generally thought of as a way to solve duplicate content problems, it can be extremely wise to use it on every (unique) page of a site to help prevent any query strings, session IDs, scraped versions, licensing deals or future developments to potentially create a secondary version and pull link juice or other metrics away from the original. We believe the canonical URL tag is a best practice to help prevent future problems, even if nothing is specifically duplicate/problematic today. Please give example.
On-Page Optimization | | AllIsWell0 -
Redirecting URLS on windows
Could anyone help out here please. A client of ours have reveloped their website from HTML to ASP (helpful!). They have 60 odd pages indexed in Google with the .html extension. We need to do a redirect on these pages so that all link juice is passed to the new pages. What would be the best way to do this please?
On-Page Optimization | | Grumpy_Carl0 -
How to avoid product's lists from making your site's content duplicated?
Hi there! We at Outitude, recently launched an outdoor activities marketplace and to make it easy for users to compare activities we show a list of available activities in each activity view. The problem is that though the content is different, the first half is practically identical. Example:
On-Page Optimization | | alexmc
Sailing for a full day: http://outitude.com/en/sailing/world/sailing-full-day and sailing for half a day: http://outitude.com/en/sailing/world/sailing-half-day both URL's are different, their content is different but most of it is not (first half of the page), so that the user can compare the activity it is currently seing with others. Questions: How can we show the activities list without it ruining the page rank? Do you advise the use of "", "" surrounding the duplicated content aka activities lists? Thanks in advance.0