Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
What countries does Google crawl from? Is it only US or do they crawl from Europe and Asia, etc.?
-
Where does Google crawl the web from? Is it in the US only, or do they do it from a European base too? The reason for asking is for GeoIP redirection. For example, if a website is using GeoIP redirection to redirect all US traffic to a .com site and all EU traffic to a .co.uk site, will Google ever see the .co.uk site?
-
Hi Keith,
In my experience Google mainly crawls from the US.
You're quite right, GeoIP redirection can cause major issues with indexation - as if you're redirecting everything from the US to the .com googlebot can't see the .co.uk site.
As such, I'm not a fan. Rather than implementing a hard redirect I prefer Amazon's approach. If you visit the amazon.com site from a UK IP you get a javascript overlay that invites you to visit the .co.uk version of the site instead - they let the user decide which site to view rather than actually redirecting them.
This is a nice solution as it ensures that the search bots can crawl both versions of the site, and rankings aren't endangered.
I hope this helps,
Hannah
-
Keith, I am having the same issue and I agree with you. The fact that Google has data centers in Europe does not necessarily mean the algos are indexing from there. I also want to set Europe and US GeoIP redirection. It would be great to get Mozers opinions on this. Hopefully this post gets freshly reviewed
-
Interesting question - I'd quite like to know what happens here too.
Matt Cutts recently posted a video on cloaking (http://youtu.be/QHtnfOgp65Q) saying that as long as you don't do anything 'special' for Googlebot you're OK, but presumably if you are redirecting IP's based on location and you don't want to prevent Googlebot form accessing your site then effectively you have to do something 'special' for Googlebot (e.g. you're doing one thing for everyone else and a different thing for Googlebot).
-
Hi,
The data center locations is interesting, but it isn't what I was looking for. I need to know whether Google crawls the web from any IP other than US IPs.
To clear up the second question, let me be more specific:
Let's say Google is crawling a .co.uk site from a US IP address. The site is using GEO IP redirection to redirect all US traffic to the .com site. Therefore, when Google attempts to crawl the .co.uk from the US IP address site it will be redirected to the .com site, never seeing the .co.uk site. Can anyone confirm that this is what happens?
-
I found an article from 2008 that shows Google data centre locations: http://bit.ly/mONhf9
Your other question is a bit confusing. Why Google wouldn't see the UK site?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Worldwide and Europe hreflang implementation.
Hi Moz ! We're having quite a discussion here and I'd like to have some inputs. Let me explain the situation and what we plan to do so far. One of our client has two separate markets : World and Europe. Both pages versions will be mostly the same, except for the fact that they will have their own products. So basically, we'd want to show only the European EN version to Europe and the standard EN version to the rest of the world, same goes for FR and ES. As far as IT, DE, CS and SK, they will only be present within the european version. Since we cannot target all Europe with a single hreflang tag, we might have to do it for every single european countries. Regarding this subject, SMX Munich recently had quite an interesting session about this topic with a confirmation coming from John Mueller saying that we can target a single URL more than once with different hreflang tags. You can read more here : http://www.rebelytics.com/multiple-hreflang-tags-one-url/ So having all this in mind, here's the implementation we plan to do : www.example.com/en/ Self canonical www.example.com/fr/ - hreflang = fr www.example.com/es/ - hreflang = es www.example.eu/it/ - hreflang = it www.example.eu/de/ - hreflang = de www.example.eu/cs/ - hreflang = cs www.example.eu/sk/ - hreflang = sk www.example.eu/fr/ - hreflang = be-fr www.example.eu/fr/ - hreflang = ch-fr www.example.eu/fr/ - hreflang = cz-fr www.example.eu/fr/ - hreflang = de-fr www.example.eu/fr/ - hreflang = es-fr www.example.eu/fr/ - hreflang = fr-fr www.example.eu/fr/ - hreflang = uk-fr www.example.eu/fr/ - hreflang = gr-fr www.example.eu/fr/ - hreflang = hr-fr etc… . This will be done for all european countries (FR, EN and ES). www.example.com/en/ - x-default Let me know what you guys think. Thanks!
International SEO | | Netleaf.ca0 -
Targeting Countries in the Middle East
Hi guys, I have a client based in the Middle East using a generic top level domain (.com), and they want to target multiple countries in the GCC (UAE, Saudi Arabia, Kuwait, Qatar etc). I’m thinking that using the hreflang tag would be the best solution here, however the pages will mostly have the exact same content. There will only be slight changes on some pages in terms of using localised title tags [client service] followed by [targeted country], h1's and meta descriptions. Is this the correct approach? And if so should this be implemented side wide or can it be implemented on selected pages only? The site will be in English only.
International SEO | | Jbeetle0 -
Can you target the same site with multiple country HREFlang entries?
Hi, I have a question regarding the country targeting aspect of HREFLANG. Can the same site be targeted with multiple country HREFlang entries? Example: A global company has an English South African site (geotargeted in webmaster tools to South Africa), with a hreflang entry targeted to "en-za", to signify English language and South Africa as the country. Could you add entries to the same site to target other English speaking South African countries? Entries would look something like this: (cd = Congo, a completely random example) etc... Since you can only geo-target a site to one country in WMT would this be a viable option? Thanks in advance for any help! Vince
International SEO | | SimonByrneIFS0 -
If I redirect based on IP will Google still crawl my international sites if I implement Hreflang
We are setting up several international sites. Ideally, we wouldn't set up any redirects, but if we have to (for merchandising reasons etc) I'd like to assess what the next best option would be. A secondary option could be that we implement the redirects based on IP. However, Google then wouldn't be able to access the content for all the international sites (we're setting up 6 in total) and would only index the .com site. I'm wondering whether the Hreflang annotations would still allow Google to find the International sites? If not, that's a lot of content we are not fully benefiting from. Another option could be that we treat the Googlebot user agent differently, but this would probably be considered as cloaking by the G-Man. If there are any other options, please let me know.
International SEO | | Ben.JD0 -
International SEO Subfolders / user journey etc
Hi According to all the resources i can find on Moz and elsewhere re int seo, say in the context of having duplicate versions of US & UK site, its best to have subfolders i.e. domain.com/en-gb/ & domain.com/en-us/ however when it comes to the user journey and promoting web address seems a bit weird to say visit us at: domain.com/en-us/ !? And what happens if someone just enters in domain.com from the US or UK ? My client wants to use an IP sniffer but i've read thats bad practice and should employ above style country/language code instead, but i'm confused about both the user journey and experience in the case of multiple sub folders. Any advice much appreciated ? Cheers Dan
International SEO | | Dan-Lawrence0 -
Working with country specific domain names vs. staying with .com
I've recently inherited a client that has a country specific domain for Canada (.ca) but there is also a US branch for the company at the .com address. They have a direct competitor that operates also in the U.S. and Canada that has decided to operate entirely under the .com address and re-direct all .ca traffic to their .com address. When I compare the link analysis data for both the .ca, .com, and competitors site, I'm finding there is a huge difference between the .ca site and the competitors site, but not a huge difference between the .com site and the competitors site. For example, the domain authorities are as follows: myclient.ca (Canadian branch) - 22 myclient.com (US branch) - 46 competitor.com - 53 When I do a brand search for my client in Canada, the Canadian branch website shows up first, but the American one is second. At this point, would it be better for my client to consolidate the two branches into the .com address and focus on increasing external followed links to the .com website? Or, is there merit in continuing to create a separate inbound link strategy for the .ca site? Thanks.
International SEO | | modernmusings0 -
Website Target in Europe
Hi, I am planning a site to target in Europe and I expect to translate my site into ten different languages namely English, French, Spanish, Italian, German, Russian, Greek, Portuguese, Dutch and Swedish. I am doing some study of this case in targeting different countries for SEO, most of the advise are the following: a. Build 10 different websites and target different geographical location in Google Webmaster b. Get 10 different country specific domains for 10 different websites I would like to hear any suggestion if there is anything better than this ? I had all the materials and translation ready but building 10 different websites or getting 10 different domains are very time consuming and costly. I would be appreciated if any one had any advise for me to make the website more management friendly. Thank you. Tom
International SEO | | Stevejobs20110 -
Country name displayed after domain name in google SERP
our online shop targets clients in the US and worldwide (same URL - no subdirectories - currency changes based on IP). when searching in google.ie or google.no for our site google displays in the SERPS "US" or "United States" after the URL for our site, but for most other US competitors it does not show the country in the SERPS. I deleted our google places listing 2 weeks ago, since I suspected it may be related, but no change so far. In google webmaster tools we have targeted the shop domain to United States, which may be another factor. Unfortunately we can not undo this setting since without it our google US ranking for the most relevant competitive keyword drops from position 8 to position 100+. Server location is in Germany which despite lots of US links and US contact info and USD currency appparently makes google think that the site is not targeting the US. Does anybody know what triggers the country name in the SERPS (google places or webmaster tools or other) and can give advice if there is any way to get rid of it.
International SEO | | lcourse0