What countries does Google crawl from? Is it only US or do they crawl from Europe and Asia, etc.?
-
Where does Google crawl the web from? Is it in the US only, or do they do it from a European base too? The reason for asking is for GeoIP redirection. For example, if a website is using GeoIP redirection to redirect all US traffic to a .com site and all EU traffic to a .co.uk site, will Google ever see the .co.uk site?
-
Hi Keith,
In my experience Google mainly crawls from the US.
You're quite right, GeoIP redirection can cause major issues with indexation - as if you're redirecting everything from the US to the .com googlebot can't see the .co.uk site.
As such, I'm not a fan. Rather than implementing a hard redirect I prefer Amazon's approach. If you visit the amazon.com site from a UK IP you get a javascript overlay that invites you to visit the .co.uk version of the site instead - they let the user decide which site to view rather than actually redirecting them.
This is a nice solution as it ensures that the search bots can crawl both versions of the site, and rankings aren't endangered.
I hope this helps,
Hannah
-
Keith, I am having the same issue and I agree with you. The fact that Google has data centers in Europe does not necessarily mean the algos are indexing from there. I also want to set Europe and US GeoIP redirection. It would be great to get Mozers opinions on this. Hopefully this post gets freshly reviewed
-
Interesting question - I'd quite like to know what happens here too.
Matt Cutts recently posted a video on cloaking (http://youtu.be/QHtnfOgp65Q) saying that as long as you don't do anything 'special' for Googlebot you're OK, but presumably if you are redirecting IP's based on location and you don't want to prevent Googlebot form accessing your site then effectively you have to do something 'special' for Googlebot (e.g. you're doing one thing for everyone else and a different thing for Googlebot).
-
Hi,
The data center locations is interesting, but it isn't what I was looking for. I need to know whether Google crawls the web from any IP other than US IPs.
To clear up the second question, let me be more specific:
Let's say Google is crawling a .co.uk site from a US IP address. The site is using GEO IP redirection to redirect all US traffic to the .com site. Therefore, when Google attempts to crawl the .co.uk from the US IP address site it will be redirected to the .com site, never seeing the .co.uk site. Can anyone confirm that this is what happens?
-
I found an article from 2008 that shows Google data centre locations: http://bit.ly/mONhf9
Your other question is a bit confusing. Why Google wouldn't see the UK site?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How well does Google's "Locale-aware crawling by Googlebot" work?
Hello, In January of this year Google introduced "Locale-aware crawling by Googlebot." https://support.google.com/webmasters/answer/6144055?hl=e Google uses different crawl settings for sites that cannot have separate URLs for each locale. ......... This is basically for sites that dynamically render contend on the same URL depending on the locale and language (IP) of the visitor. If e.g. a visitor was coming from France, the targeted page would load in french. If a visitor was coming from the US the same page would load in English on the same URL. Does anyone have any experience with this setup and how well it works? How well do the different versions of a page get indexed, and how well do those pages rank? In the example above, does the french content get indexed correctly? Many thanks!
International SEO | | Veva0 -
Google does not index UK version of our site, and serves US version instead. Do I need to remove hreflanguage for US?
Webmaster tools indicates that only 25% of pages on our UK domain with GBP prices is indexed.
International SEO | | lcourse
We have another US domain with identical content but USD prices which is indexed fine. When I search in google for site:mydomain I see that most of my pages seem to appear, but then in the rich snippets google shows USD prices instead of the GBP prices which we publish on this page (USD price is not published on the page and I tested with an US proxy and US price is nowhere in the source code). Then I clicked on the result in google to see cached version of page and google shows me as cached version of the UK product page the US product page. I use the following hreflang code: rel="alternate" hreflang="en-US" href="https://www.domain.com/product" />
rel="alternate" hreflang="en-GB" href="https://www.domain.co.uk/product" /> canonical of UK page is correctly referring to UK page. Any ideas? Do I need to remove the hreflang for en-US to get the UK domain properly indexed in google?0 -
How to get EU visitors to .co.uk and US visitors to .com
Hi all Re: www.explanar.com and www.explanar.co.uk We have developed a website for the US and UK, with a shop on each. Each site has a network of countries they can export to. US sells and ships to US, South America, Caribbean and so on UK sells and ships to UK, Europe, and Africa, etc Currently, visitors from Germany (.de) searching for phrases "Explanar", or "golf swing trainer" get the .com (US) site. They cannot buy from this site, so we have added a message to tell them to go to the UK site (.co.uk, but what I really want is for the European visitors to find the UK site only. .com is the original URL, but with new content
International SEO | | Crumpled_Dog
.co.uk has just been set up In the long run, we will set up .de URLs with German language, and other countries will follow, but not all. Any thoughts would be great. Thanks
Scott0 -
Multiple geographic targeting in Europe
I have a site that is applicable to German speaking people in central europe. If I were to geotarget the site in google webmaster tools to Germany, would that prevent users in Switzerland or Austria seeing the site in their search results.
International SEO | | zeropointlabs0 -
Delivering different content according to country
Hey, I have a question regarding different content according to country (IP)-
International SEO | | Kung_fu_Panda
We planing to serve mobile users using dynamic HTML serving (on the same url)
Is it possible to serve different content for different devices + different IPs (for example different content for a user from US android and someone from UK android ) thanks!0 -
Targeting an Specific Country Audience - Domain Q
Hiya everyone! I know this might entail a novice SEO question, but i am having some doubts. Hope you can give your opinions. Its kind of technical question regarding domain and country targeting. I have a Steel Construction company targeting only the audience of the particular country. Last year, i bought the targeted domains for my brand (company name), as in domain.country specific TLD, Should i use these domains, redirect them, or something? Would that help?. I currently use domain.com, but i am constantly being beaten down by websites with domain specific with overly thin content, no PA or DA, and 0 links to their site. Should i use my country specific domains, would that make a difference? Note: I also run some marketing campaigns for charitable foundation i started, and i used country specific domain and server, and with little effort i ranked top 3 in most of the desired terms. Any help or comment is appreciated, Thanks!
International SEO | | JesusD0 -
Working with country specific domain names vs. staying with .com
I've recently inherited a client that has a country specific domain for Canada (.ca) but there is also a US branch for the company at the .com address. They have a direct competitor that operates also in the U.S. and Canada that has decided to operate entirely under the .com address and re-direct all .ca traffic to their .com address. When I compare the link analysis data for both the .ca, .com, and competitors site, I'm finding there is a huge difference between the .ca site and the competitors site, but not a huge difference between the .com site and the competitors site. For example, the domain authorities are as follows: myclient.ca (Canadian branch) - 22 myclient.com (US branch) - 46 competitor.com - 53 When I do a brand search for my client in Canada, the Canadian branch website shows up first, but the American one is second. At this point, would it be better for my client to consolidate the two branches into the .com address and focus on increasing external followed links to the .com website? Or, is there merit in continuing to create a separate inbound link strategy for the .ca site? Thanks.
International SEO | | modernmusings0 -
Moving British site to the US... who will have .com? US or UK?
We are the UK's first baby social commerce site launched in Nov 2011. We're doing quite well and are looking at expanding to the US. However I'm not sure what advice you'd give me in terms of internationalising the site. I see three options on how to deal with the URL structure? Make US site as .com as it will be my main source of revenue for the long run and redirect all British traffic to .co.uk Have .com for both UK and US but have the URL as either: us.babyhuddle.com or as babyhuddle.com/us/. Same thing for the UK Another option? Would love to hear the feedback from you guys. Thanks, Walid
International SEO | | walidalsaqqaf0