What countries does Google crawl from? Is it only US or do they crawl from Europe and Asia, etc.?
-
Where does Google crawl the web from? Is it in the US only, or do they do it from a European base too? The reason for asking is for GeoIP redirection. For example, if a website is using GeoIP redirection to redirect all US traffic to a .com site and all EU traffic to a .co.uk site, will Google ever see the .co.uk site?
-
Hi Keith,
In my experience Google mainly crawls from the US.
You're quite right, GeoIP redirection can cause major issues with indexation - as if you're redirecting everything from the US to the .com googlebot can't see the .co.uk site.
As such, I'm not a fan. Rather than implementing a hard redirect I prefer Amazon's approach. If you visit the amazon.com site from a UK IP you get a javascript overlay that invites you to visit the .co.uk version of the site instead - they let the user decide which site to view rather than actually redirecting them.
This is a nice solution as it ensures that the search bots can crawl both versions of the site, and rankings aren't endangered.
I hope this helps,
Hannah
-
Keith, I am having the same issue and I agree with you. The fact that Google has data centers in Europe does not necessarily mean the algos are indexing from there. I also want to set Europe and US GeoIP redirection. It would be great to get Mozers opinions on this. Hopefully this post gets freshly reviewed
-
Interesting question - I'd quite like to know what happens here too.
Matt Cutts recently posted a video on cloaking (http://youtu.be/QHtnfOgp65Q) saying that as long as you don't do anything 'special' for Googlebot you're OK, but presumably if you are redirecting IP's based on location and you don't want to prevent Googlebot form accessing your site then effectively you have to do something 'special' for Googlebot (e.g. you're doing one thing for everyone else and a different thing for Googlebot).
-
Hi,
The data center locations is interesting, but it isn't what I was looking for. I need to know whether Google crawls the web from any IP other than US IPs.
To clear up the second question, let me be more specific:
Let's say Google is crawling a .co.uk site from a US IP address. The site is using GEO IP redirection to redirect all US traffic to the .com site. Therefore, when Google attempts to crawl the .co.uk from the US IP address site it will be redirected to the .com site, never seeing the .co.uk site. Can anyone confirm that this is what happens?
-
I found an article from 2008 that shows Google data centre locations: http://bit.ly/mONhf9
Your other question is a bit confusing. Why Google wouldn't see the UK site?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Problem to get multilingual posts indexed on Google
Last year on June I decided to make my site multi-lingual. The domain is: https://www.dailyblogprofits.com/ The main language English and I added Portuguese and a few posts on Spanish. What happened since then? I started losing traffic from Google and posts on Portuguese are not being indexed. I use WPML plugin to make it multi-lingual and I had Yoast installed. This week I uninstalled Yoast and when I type on google "site:site:dailyblogprofits.com/pt-br" I started seeing Google indexing images, but still not the missing posts. I have around 145 posts on Portuguese, but on Search Console it show only 57 hreflang tags. Any idea what is the problem? I'm willing to pay for an SEO Expert to resolve this problem to me.
International SEO | | Cleber0090 -
Web Site Migration - Time to Google indexing
Soon we will do a website migration .com.br to .com/pt-br. Wi will do this migration when we have with lower traffic. Trying to follow Google Guidelines, applying the 301 redirect, sitemap etc... I would like to know, how long time the Google generally will use to transfering the relevance of .com.br to .com/pt-br/ using redirect 301?
International SEO | | mobic0 -
How is Google determining location for geotargeting
Hello, I have a question related to Geotargeting.
International SEO | | Lvet
Let's suppose I have a website: mysite.co.uk
As far as I understand Google will consider this site targeted for the UK and will appear on searches preferentially from the UK. What happen now if I have a person located in Spain (with a Spanish IP) and searching in Google.co.uk. Will mysite.co.uk still appear in searches? What are the factors that Google takes into account for Geotargeting? Thank you0 -
Ranking UK company in Google.com
Hi all, I have a UK client with a .com domain, hosted on a US server, but the physical business premises is based in the UK. Their product is a really great product and available for export to the US. I want to rank them higher in the US, more specifically Google.com. I've helped them rank very well organically in the UK (google.co.uk) for some great terms, however they rank almost nowhere in google.com (gl=us) for the same terms, for example: In Google.co.uk they rank #3 for the key-phrase.
International SEO | | seowoody
In Google.com they rank #90 for the same key-phrase. I've got them some great US focused links with PR coverage including MSN Cars, nydailynews.com etc. I just wondered if there was any one "golden ticket" for boosting US rankings? I've read that a physical business premises located in the US helps a lot. Can anyone confirm this and if so, would a rented PO box in the US help? The site has great social signals too, growing twitter following and many FB likes/shares etc. Any other tips/advice? Thanks in advance,
Cheers,
Woody 🙂0 -
Country-specific SEO
Hi, my client offers courses that whilst based in Manchester, England are mainly attended by people in countries such as Georgia, Libya and Nigeria etc. The people that attend the courses are fluent in English. We're looking at performing country-specific SEO and I have a few queries. The plan so far: Obtain TLD's in each target country. These TLD's would be hosted on the same server as the core site based in England. Option 1: Each TLD would be a microsite with content specific to the country including geo-signals, in English. Option 2: Each TLD would 301 redirect to the core site, i.e. example.sa redirects to example.com/sa/ and this country-specific section would have relevant geo-signals. So I have 3 questions at this point: 1. Most-all of the tips I have seen about country-specific SEO assume that the content should be translated to the native tongue although in this case, the audience are fluent English speaks. Does this make a difference? Is it okay to use English and still be able to rank in country-specific search engines? 2. Between options 1 & option 2 - which would be the optimum setup? 3. Last question, if we obtain the TLD's I hear that it's not necessary to also host that TLD in the target country, is this right? Thanks.
International SEO | | lokito0 -
Google search cache points to and uses content from different url
We have two sites, 1 in new zealand: ecostore.co.nz and 1 in Australia: ecostoreaustralia.com.au Both sites have been assigned with the correct country in Webmaster tools Both site use the same urls structure and content for product and category pages Both sites run off the same server in the US but have unique ip adresses. When I go to google.com.au and search for: site:ecostoreaustralia.com.au I get results which google says are from the Australian domain yet on closer inspection it is actually drawing content from the NZ website. When I view a cached page the URL bar displays the AU domain name but on the page (in the top grey box) it says: _This is Google's cache of http://www.ecostore.co.nz/pages/our-highlights. _ Here is the link to this page: http://webcache.googleusercontent.com/search?q=cache:Zg_CYkqyjP4J:www.ecostoreaustralia.com.au/pages/our-highlights+&cd=7&hl=en&ct=clnk&gl=au In the last four weeks the ranking of the AU website has dropped significantly and the NZ site now ranks first in Google AU, where before the AU site was listed first. Any idea what is going wrong here?
International SEO | | ArchMedia0 -
How to replace my .co.uk site with my .com site in the US Google results
My customer and I are based in the UK. My customer's site, www.blindbolt.co.uk has been around for years. Last year we launched their American site, www.blindboltusa.com. Searching on google.com (tested both via proxy and using the gl=us querystring trick), a search for blind bolt on the US Google returns our www.blindbolt.co.uk site. We would like it to show our www.blindboltusa.com website in US searches. Webmaster tools has the Geographic Target set correctly for each site. Does anyone have any ideas or suggestions please? Thanks.
International SEO | | OffSightIT0 -
Same language many countries
Hello, I live in Belgium and in this country you've 3 languages : french, dutch and german. I've customers from many countries : France, Nederlands,... and for my website in ".be" (we'll say www.mysite.be for example) I've choosen the french language. My question is can I've the same content on my site : www.mysite.be and www.mysite.fr without duplicate content or should I forgot using www.mysite.fr to avoid the D.C. problem? And with my site : www.mysite.be should I've more difficult to rank in France for example? Thank you for your answer, Jonathan
International SEO | | JonathanLeplang0