International Sites and Duplicate Content
-
Hello,
I am working on a project where have some doubts regarding the structure of international sites and multi languages.Website is in the fashion industry. I think is a common problem for this industry. Website is translated in 5 languages and sell in 21 countries.
As you can imagine this create a huge number of urls, so much that with ScreamingFrog I cant even complete the crawling.
Perhaps the UK site is visible in all those versions
http://www.MyDomain.com/en/GB/
http://www.MyDomain.com/it/GB/
http://www.MyDomain.com/fr/GB/
http://www.MyDomain.com/de/GB/
http://www.MyDomain.com/es/GB/
Obviously for SEO only the first version is important
One other example, the French site is available in 5 languages and again...
http://www.MyDomain.com/fr/FR/
http://www.MyDomain.com/en/FR/
http://www.MyDomain.com/it/FR/
http://www.MyDomain.com/de/FR/
http://www.MyDomain.com/es/FR/
And so on...this is creating 3 issues mainly:
-
Endless crawling - with crawlers not focusing on most important pages
-
Duplication of content
-
Wrong GEO urls ranking in Google
I have already implemented href lang but didn't noticed any improvements. Therefore my question is
Should I exclude with "robots.txt" and "no index" the non appropriate targeting?
Perhaps for UK leave crawable just English version i.e. http://www.MyDomain.com/en/GB/, for France just the French version http://www.MyDomain.com/fr/FR/ and so on
What I would like to get doing this is to have the crawlers more focused on the important SEO pages, avoid content duplication and wrong urls rankings on local Google
Please comment
-
-
Hey Guido, don't know if it's the best solution, but could be a temporary fix until the best solution is in place. I suggest to move forward with proper HREF LANG tagging or definitely delete those irrelevant languages. Try to do what I said before about validate each country/language and submit a sitemap.xml reflecting that folder to see crawl and index stats pero country/language. Add a sitemap index and obviously validate your entire domain. Just block in the robots.txt unnecessary folders, like images, js libraries, etc. to save crawl budget to your domain.
Let me know if you have another doubt
-
Thank you Antonio, insightful and clear.
There is really not a need of EN versions of localized sites, I think has been done more as was easier to implement (original site is EN-US).
Don't you think robots and noindex EN version of localized sites could be the best solution? for sure is the easier one to implement without affecting UX.
-
Don't know why you have a UK oriented site for German and Italian people, I think is not important those languages in a country mainly English speaking (not US for example, there you must have a Spanish version, or in Canada for English and French). The owner must have their reasons.
Besides this, about your questions:
- If those non-relevant languages must live there, it's correct to implement HREF LANG (may take some time to show results). Also, if the domain is gTLD, you can validate all the subfolders in Google Search Console and choose the proper International targeting. With the ammount of languages and countries I imagine this might be a pain in the ***.
- About the crawling, for large sitesI recommend to crawl per language. If neccesary, per language-country. In this instance I recommend to create a sitemap XML per language or language-country for just HTML pages (hopefully dynamically updated by the e-commerce), create a Sitemap Index in the root of the domain and submit them in Google Search Console (better if you validated the languages or language-country). With this you can answer the question if some language or country are being not indexed with the Submited/Indexed stadistics of GSC.
- Maybe the robots.txt might save your crawl budget, but I'm not a fan of de-index if those folders are truly not relevant (after all, there should be a italian living in UK. If you can't delete the irrelevant langauges for some countries, this can be an option
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Setting up international site subdirectories in GSC as separate properties for better geotargeting?
My client has an international website with a subdirectory structure for each country and language version - eg. /en-US. At present, there is a single property set up for the domain in Google Search Console but there are currently various geotargeting issues I’m trying to correct with hreflang tags. My question is, is it still recommended practise and helpful to add each international subdirectory to Google Search Console as an individual property to help with correct language and region tagging? I know there used to be properly sets for this but haven’t found any up to date guidance on whether setting up all the different versions as their own properties might help with targeting. Many thanks in advance!
International SEO | | MMcCalden0 -
International Site Merge
Hello, I've never had to deal with an international site before, let alone a site merge. These are two large sites, we've got a few smaller old sites that are currently redirecting to the main site (UK). We are looking at moving all the sites to the .com domain. We are also currently not using SSL (on the main pages, we are on the checkout). We also have a m.domain.com site. Are there any good guides on what needs to be done? My current strategy would be: Convert site to SSL. Mobile site and desktop site must be on the same domain. Start link building to the .com domain now (weaker link profile currently) What's the best way of handling the domains and languages? We're currently using a .tv site for the UK and .com for the US. I was thinking, and please correct me if i'm wrong, that we move the US site from domain.com to domain.com/us/ and the domain.tv to domain.com/en/ Would I then reference these by the following: What would we then do with the canonicals? Would they just reference their "local" version? Any advice or articles to read would really be appreciated.
International SEO | | ThomasHarvey0 -
International SEO Proposal
Hi, I need to create an international seo proposal and wondered what are the best bits of international SEO I should include? I have been reading up on loads of blogs wondered if anyone had some great ideas 🙂 Much appreciated.
International SEO | | karl621 -
SEO Strategy for international website with similar content
Hello, If a company is in different countries and has same content in most of the countries does it hurt SEO? For Ex. fibaro.com is the website that I am researching and I have seen the indexed pages to be about 40,000 however there is not much content on it. On further inspection I noticed that for every country the sub folder is different. So for us it will be fibaro.com/us/motion-sensor and for Europe fibaro.com/en/motion-sensor. Now both of these pages have same content on it and the company is in 86 countries so imagine the amount of duplicate content it has. Does anybody have any ideas on what should be an ideal way to approach this? Thanks
International SEO | | Harveyspecter0 -
Website Domains, Geographical targeting and Duplicate Content
My colleagues in Holland have 2 websites. I've copied and pasted their question - my comments are at the bottom "www.ancoferwaldram.nl with NL, EN and FR language www.ancoferwaldram.com with only EN language The EN versions Google sees as “duplicate content” so we have to get rid of that. I think we better use 1 website: www.ancoferwaldram.com with NL, EN, FR and maybe other languages and deactivate www.ancoferwaldram.nl Or keep the www.ancoferwaldram.nl with only the NL language? Or keep the www.ancoferwaldram.nl with direct links to www.ancoferwaldram.com and no content?" The focus is to get the site to rank in Non-eu countries for export. So given the .nl has higher DA (though only about 15) would it be better to have seperate .fr, .be, .com sites for specific languages and geo targeting. Or would it be better to keep everything on the same site? If so which domain? i assume that the duplicate content can be resolved by stating which is the canonical version, once the domain strategy is resolved welcome any thoughts here. 🙂
International SEO | | Zippy-Bungle0 -
E-Commerce site in 2 languages - Duplicate content or not?
How does Google view this? Our current site works like:
International SEO | | bjs2010
www.domain.com/EN - English
www.domain.com/ES - Spanish All products are the same, just different language and different URL for them - is this good or bad? I thought of either Going with .co.uk or .com for "English" and a .es for "Spanish"
OR Subdomaining it. www.es.domain.com and www.en.domain.com Any advice appreciated!0 -
I have on site translated into several languages on different TLDs, .com, .de, .co.uk, .no, etc. Is this duplicate content?
Three of the sites are English (.co.uk, .com, .us) as well as foreign (.de, .no, etc.) - are these all seen as having duplicate content on every site? They're hosted under the same EpiServer backend system if this helps. But I am still copying and pasting content over each site, and translating where necessary, so I'm concerned this is indexed as being large amounts of duplicate content. Site traffic doesn't appear to be suffering but as I'm currently putting together new SEOs strategies, I want to cover this possibility. Any advice on ensuring the sites aren't penalised appreciated!
International SEO | | hurtigruten0 -
How to optimise a site for 2 countries
Hi there - Any help with the below much appreciated I am helping an Australian company, producing packaging products for businesses. Their site is hosted in Australia and their offices are in Australia. They have asked me to take care of both on-page and off-page SEO so that they rank for keywords related to their products - e.g. 'cardboard boxes'. This should be fairly straightforward for Australian based (.com.au) searchers, but they also supply their products to South Africa, and so want their results to show up also for South African based (.co.za) searchers. Also consider: it is not typical for searchers for these products to use geomodifiers in their search terms there is no unique content for the South African market versus the Australian... the product information is essentially identical. What should we do to ensure their results show up equally for those in South Africa as well as Australia? I am considering building a completely separate site, hosted in South Africa and specifically for the S.A market, but will the duplicate content effect be an issue? Also, this would essentially mean double the SEO effort, is there no way I could achieve our goals more efficiently? many thanks to any help
International SEO | | dnaynay0