Duplicate content on URL trailing slash
-
Hello,
Some time ago, we accidentally made changes to our site which modified the way urls in links are generated. At once, trailing slashes were added to many urls (only in links).
Links that used to send to
example.com/webpage.htmlWere now linking to
example.com/webpage.html/Urls in the xml sitemap remained unchanged (no trailing slash).
We started noticing duplicate content (because our site renders the same page with or without the trailing shash). We corrected the problematic php url function so that now, all links on the site link to a url without trailing slash.
However, Google had time to index these pages. Is implementing 301 redirects required in this case?
-
Yes you want to have it match the canonical tag so most effective method is to 301 redirect so they match the canonical tag site map and robots.txt etc. You can use a Regex code like this at the end of the URL /?$ in the case of category URLs it will allow them when needed.
if you use the proper 301 you will not have to deal with the category issue anyway.
rel="canonical" href="https://moz.com/community/q/duplicate-content-on-url-trailing-slash" />
I hope this is able to shed more light on the issue and great answer Eric.
Hope I was of help,
Tom
-
Hi Eric,
I was at Step 3 of your 3 Step plan, looking for confirmation as to whether or not the 301 redirects were required in this situation.
Thanks!
-
Hi yacpro13! Did Eric or Thomas answer your question, and if so, would you mind marking one or both responses as a "Good Answer?"
Otherwise, what questions do you still have?
-
If you have changed the URLs with trailing slashes, then there are a few things you'll want to do:
-
make sure all the internal links on your site are updated to point to the proper version.
-
make sure that the sitemap.xml file(s) are correct, pointing to the proper version.
-
set up 301 permanent redirects so that the ones with the slash are redirecting to the old URLs.
As long as you have corrected the links internally, updated the sitemap file, and set up the 301 redirects, everything should go "back to normal" within a fairly short period of time. You will need to give it time, though, as Google will need to re-crawl all of those URLs and get it all ironed out.
-
-
I have provided the Apache and Nginx configurations you would need in addition to a URL that will convert
Apache Htaccess to Nginx
The instructions are right here
Remove Trailing Slash
Just like with the WWW example, some prefer to remove the trailing slash. It's a commonly debated question that you'll find around the Internet, but it just depends on what you prefer.
Remember, though, your browser and even your server, by default, add a trailing slash to a directory. It is done for a reason. If you must strip the trailing slash, though, this is how you would do it:
<code class="hljs apache">RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_URI} !(.*)$ RewriteRule ^(.*)$ http://www.domain.com/$1 [R=301,L]</code>
For Nginx
nginx configuration location ~ (.)$ { } location / { if (!-e $request_filename){ rewrite ^(.)$ http://www.domain.com/$1 redirect; } }
The explanation for this rule is the same as it is for when we want to add a trailing slash, just in reverse. We can also specify specific directories that we don't want apply this rule over.
<code class="hljs apache">RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_URI} !directory/(.*)$ RewriteCond %{REQUEST_URI} !(.*)$ RewriteRule ^(.*)$ http://www.domain.com/$1 [R=301,L]</code>
For Nginx
nginx configuration location ~ directory/(.)$ { } location ~ (.)$ { } location / { if (!-e $request_filename){ rewrite ^(.*)$ http://www.domain.com/$1 redirect; } }
Please see the note about mod_dir and the
DirectorySlash
directive in the previous example. You might need to turn this directive off.HTaccess converter for Apache to Nginx configuration.
http://winginx.com/en/htaccess
https://www.maxcdn.com/one/tutorial/remove-trailing-slash/
https://www.crucialhosting.com/knowledgebase/htaccess-apache-rewrites-examples
https://moz.com/community/q/how-to-remove-trailing-slashes-in-urls-using-htaccess-apache
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEM Rush & Duplicate content
Hi SEMRush is flagging these pages as having duplicate content, but we have rel = next etc implemented: https://www.key.co.uk/en/key/brand/bott https://www.key.co.uk/en/key/brand/bott?page=2 Or is it being flagged as they're just really similar pages?
Intermediate & Advanced SEO | | BeckyKey0 -
Cross Domain duplicate content...
Does anyone have any experience with this situation? We have 2 ecommerce websites that carry 90% of the same products, with mostly duplicate product descriptions across domains. We will be running some tests shortly. Question 1: If we deindex a group of product pages on Site A, should we see an increase in ranking for the same products on Site B? I know nothing is certain, just curious to hear your input. The same 2 domains have different niche authorities. One is healthcare products, the other is general merchandise. We've seen this because different products rank higher on 1 domain or the other. Both sites have the same Moz Domain Authority (42, go figure). We are strongly considering cross domain canonicals. Question 2 Does niche authority transfer with a cross domain canonical? In other words, for a particular product, will it rank the same on both domains regardless of which direction we canonical? Ex: Site A: Healthcare Products, Site B: General Merchandise. I have a health product that ranks #15 on site A, and #30 on site B. If I use rel=canonical for this product on site B pointing at the same product on Site A, will the ranking be the same if I use Rel=canonical from Site A to Site B? Again, best guess is fine. Question 3: These domains have similar category page structures, URLs, etc, but feature different products for a particular category. Since the pages are different, will cross domain canonicals be honored by Google?
Intermediate & Advanced SEO | | AMHC1 -
Duplicated Content with Index.php
Good Afternoon, My website uses Joomla CMS and has the htaccess rewrite code enabled to ensure the use of search engine friendly URLs (SEF's). While browsing the crawl diagnostics I have found that Moz considers the /index.php URL a duplicate to our root. I will always under the impression that the htaccess rewrite took care of that issue and obviously I would like to address it. I attempted to create a 301 redirect from the index.php URL to the root but ran into an issue when attempting to login to the admin portion of the website as the redirect sent me back to the homepage. I was curious if anyone had advice for handling the index.php duplication issue, specifically with Joomla. Additionally, I have confirmed that in Google Webmasters, under URL parameters, the index.php parameter is set as 'Representative URL'.
Intermediate & Advanced SEO | | BrandonEML0 -
Best practice with duplicate content. Cd
Our website has recently been updated, now it seems that all of our products pages look like this cdnorigin.companyname.com/catagory/product Google is showing these pages within the search. rather then companyname.com/catagory/product Each product page does have a canaonacal tag on that points to the cdnorigin page. Is this best practice? i dont think that cdnorigin.companyname etc looks very goon in the search. is there any reason why my designer would set the canonical tags up this way?
Intermediate & Advanced SEO | | Alexogilvie0 -
Issue with duplicate content in blog
I have blog where all the pages r get indexed, with rich content in it. But In blogs tag and category url are also get indexed. i have just added my blog in seomoz pro, and i have checked my Crawl Diagnostics Summary in that its showing me that some of your blog content are same. For Example: www.abcdef.com/watches/cool-watches-of-2012/ these url is already get indexed, but i have asigned some tag and catgeory fo these url also which have also get indexed with the same content. so how shall i stop search engines to do not crawl these tag and categories pages. if i have more no - follow tags in my blog does it gives negative impact to search engines, any alternate way to tell search engines to stop crawling these category and tag pages.
Intermediate & Advanced SEO | | sumit600 -
Duplicate content question? thanks
Hi, Im my time as an SEO I have never come across the following two scenarios, I am an advocate of using unique content, therefore always suggest and in cases demand that all content is written or re-written. This is the scenarios I am facing right now. For Example we have www.abc.com (has over 200 original recipes) and then we have www.xyz.com with the recipes but they are translated into another language as they are targeting different audiences, will Google penalize for duplicate content? The other issue is that the client got the recipes from www.abc.com (that have been translated) and use them in www.xyz.com aswell, both sites owned by the same company so its not pleagurism they have legal rights but I am not sure how Google will see it and if it will penalize the sites. Thanks!
Intermediate & Advanced SEO | | M_81 -
HTTPS Duplicate Content?
I just recieved a error notification because our website is both http and https. http://www.quicklearn.com & https://www.quicklearn.com. My tech tells me that this isn't actually a problem? Is that true? If not, how can I address the duplicate content issue?
Intermediate & Advanced SEO | | QuickLearnTraining0 -
Is it fine to use an iframe for video content? Will it still be indexed on your URL?
If we host a video on a third party site and use an iframe to display it on our site, when the video is indexed in SERPs will it show on our site or on the third party site?
Intermediate & Advanced SEO | | nicole.healthline0