What countries does Google crawl from? Is it only US or do they crawl from Europe and Asia, etc.?
-
Where does Google crawl the web from? Is it in the US only, or do they do it from a European base too? The reason for asking is for GeoIP redirection. For example, if a website is using GeoIP redirection to redirect all US traffic to a .com site and all EU traffic to a .co.uk site, will Google ever see the .co.uk site?
-
Hi Keith,
In my experience Google mainly crawls from the US.
You're quite right, GeoIP redirection can cause major issues with indexation - as if you're redirecting everything from the US to the .com googlebot can't see the .co.uk site.
As such, I'm not a fan. Rather than implementing a hard redirect I prefer Amazon's approach. If you visit the amazon.com site from a UK IP you get a javascript overlay that invites you to visit the .co.uk version of the site instead - they let the user decide which site to view rather than actually redirecting them.
This is a nice solution as it ensures that the search bots can crawl both versions of the site, and rankings aren't endangered.
I hope this helps,
Hannah
-
Keith, I am having the same issue and I agree with you. The fact that Google has data centers in Europe does not necessarily mean the algos are indexing from there. I also want to set Europe and US GeoIP redirection. It would be great to get Mozers opinions on this. Hopefully this post gets freshly reviewed
-
Interesting question - I'd quite like to know what happens here too.
Matt Cutts recently posted a video on cloaking (http://youtu.be/QHtnfOgp65Q) saying that as long as you don't do anything 'special' for Googlebot you're OK, but presumably if you are redirecting IP's based on location and you don't want to prevent Googlebot form accessing your site then effectively you have to do something 'special' for Googlebot (e.g. you're doing one thing for everyone else and a different thing for Googlebot).
-
Hi,
The data center locations is interesting, but it isn't what I was looking for. I need to know whether Google crawls the web from any IP other than US IPs.
To clear up the second question, let me be more specific:
Let's say Google is crawling a .co.uk site from a US IP address. The site is using GEO IP redirection to redirect all US traffic to the .com site. Therefore, when Google attempts to crawl the .co.uk from the US IP address site it will be redirected to the .com site, never seeing the .co.uk site. Can anyone confirm that this is what happens?
-
I found an article from 2008 that shows Google data centre locations: http://bit.ly/mONhf9
Your other question is a bit confusing. Why Google wouldn't see the UK site?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is there a Risk Around Creating a Website for Each Country in The World?
Is there a risk around creating a website for each country in the world with similar to identical content depending on the language? We need to serve prices and the local currencies and be compliant with regulations. We're planning to use rel=canonical and HREFLANG tags to help with consolidation and GEO-targeting.
International SEO | | ari_seo0 -
Setting up a website targeted for the US
Hi, As an English company we have a co.uk domain with .com domain pointing to this. We are now looking to launch a separate (new) website targeting the American market and I have been asked to do the following: If an American or Canadian IP address visits the .com website it automatically goes to our newly created website i.e. website 2. If a non-American or non-Canadian IP address goes to .com it automatically goes to the original website i.e. website 1. If a user is on website 1 and clicks an American flag it takes the website user to website 2. If a user is on website 2 and clicks on the UK flag it takes the website user to website 1. Can anyone advise the best way to go about doing this as I feel that this could effect our search rankings. I am concerned how the search engines will penalize website 2 (original site) which has good rankings. Thanks in advance.
International SEO | | Cybertill0 -
Opening Up Webmaster Tools Geo-Targeting to Everyone Instead of US
Hi Mozzers, I wanted to get everyone's opinion on this. We have an e-commerce site that was targeting 3 different geo-locations. The main one is a .com. The other two were Canada and UK based. We recently turned off our Canada site and 301 to the .com. The .com site had been targeted towards United States in Google Webmaster Tools. Should I remove the Webmaster Tools geo-targeting now that Canada is pointing towards our main site? I don't want to lose rankings for US since that is our main market and the .com site is our main revenue driver and focus. I'm inclined to leave it as is because I still see our .com site ranking when I search in Google CA. From what I've seen is that it can take months for something to change and I definitely don't want anything to go awry. Just wanted to get everyone's thoughts on this. Thanks in advance.
International SEO | | SylviaH0 -
Any practical examples of ranking 1 domain in multiple countries?
Hi, I've done a fair amount of research on international SEO including here on MOZ but was hoping some fellow Mozzers might have some practical examples of how they have got 1 domain to rank in multiple countries, ideally US & UK. Im possibly looking at getting a high authority domain which ranks great on US into the UK engines. I want to keep to the 1 domain to benefit from the high authority and for logistical reasons. Thanks in advance, Andy
International SEO | | AndyMacLean0 -
If I redirect based on IP will Google still crawl my international sites if I implement Hreflang
We are setting up several international sites. Ideally, we wouldn't set up any redirects, but if we have to (for merchandising reasons etc) I'd like to assess what the next best option would be. A secondary option could be that we implement the redirects based on IP. However, Google then wouldn't be able to access the content for all the international sites (we're setting up 6 in total) and would only index the .com site. I'm wondering whether the Hreflang annotations would still allow Google to find the International sites? If not, that's a lot of content we are not fully benefiting from. Another option could be that we treat the Googlebot user agent differently, but this would probably be considered as cloaking by the G-Man. If there are any other options, please let me know.
International SEO | | Ben.JD0 -
Targeting specific Geographic areas. Use 1 large.Com or several smaller country specific TLDs?
Hi, I have a small number of exact match domains, both country specific TLDs and also the Generic TLD dot com and dot net. They are: ExactMatch**.Com**
International SEO | | Hurf
ExactMatch**.Net** ExactMatch**.Co.Uk**
ExactMatch**.Ca**
ExactMatch**.Co.Nz**
ExactMatch**.Co.Za** We have already successfully launched our UK site using the exact match .co.uk and this is currently number 2 in the UK SERPS for the Google, Yahoo and Bing. They are/will be niche specific classified ad sites, which are Geographically targeted by country (to Engish speakers in the main) and each region is likely to have a minumum of 2,000 unique listings submitted over the course of a year of so. My question (FINALLY) is this: Am I better to build one large global site (will grow to approx. 12,000 listings) using EXACTMATCH.Com with .com - targeting US users and then geo-targeted sub directories (ExactMatch.Com/Nz etc) - each sub dir targeted to the matching geographic area in webmaster tools, or use the ccTLDs and host each site in the country with perhaps (each site growing to approx 2,000 listings) I could use the ccTLDs just for marketing/branding onlyand redirect these to the specific sub directory of the .com site? I am aware that there is one main ccTLD that I cannot get .Com.Au (as I am not a resident of Australia - and it is already in use.) so I was wondering if the single site with .Com/AU/ etc might help me better target that country? If I use each ccTLD as separate sites I suppose I could use the largely redundant .net to target Australia? Your thoughts and advice would be most welcome. Thanks! An additional bit of intormation (or two) the .com is circa 2004. The product advertised is a reasonably bulky (perhaps 6kgs boxed) physical product and therefore the seller is unlikely to want to ship globally - will this make them shy away from a global site - even one divided into global sub sections? FYI Seller can specify in their listing Will Ship To ....... I would be open to looking at using the front page of the .Com site as a page which visitors select the country they wish to buy/sell on. (IF it is the general consensus that it is better to create one large site.) Consider also please how the end user is likely to percieve the benefits to them of one LARGE SITE versus TARGETED SITE - I know the .Com would be divided into geographic sub directories, but I am not sure if they won't see an additinal benefit to the ccTLD - Does this add a degree of reassurance and relevance that a .com/ccTLD cannot provide? I suppose I am biased by the fact that ebay use ccTLDs? Thanks again - and please forgive my tone which may suggest I am playing devil's advocate here. I am very torn on this issue.0 -
Multi-lingual SEO: Country-specific TLD's, or migration to a huge .com site?
Dear SEOmoz team, I’m an in-house SEO looking after a number of sites in a competitive vertical. Right now we have our core example.com site translated into over thirty different languages, with each one sitting on its own country-specific TLD (so example.de, example.jp, example.es, example.co.kr etc…). Though we’re using a template system so that changes to the .com domain propagate across all languages, over the years things have become more complex in quite a few areas. For example, the level of analytics script hacks and filters we have created in order to channel users through to each language profile is now bordering on the epic. For a number of reasons we’ve recently been discussing the cost/benefit of migrating all of these languages into the single example.com domain. On first look this would appear to simplify things greatly; however I’m nervous about what effect this would have on our organic SE traffic. All these separate sites have cumulatively received years of on/off-site work, and even if we went through the process of setting up page-for-page redirects to their new home on example.com, I would hate to lose all this hard-work (and business) if we saw our rankings tank as a result of the move. So I guess the question is, for an international business such as ours, which is the optimal site structure in the eyes of the search engines; Local sites on local TLD’s, or one mammoth site with language identifiers in the URL path (or subdomains)? Is Google still so reliant on TLD for geo targeting search results, or is it less of a factor in today’s search engine environment? Cheers!
International SEO | | linklater0 -
How can I view Google.com SERPs from outside the US?
If I go to Google.com I get redirected back to Google.co.uk and search results have a UK bias. I'm trying to research the US market and have a hazy recolection of Rand demonstrating how you can add a few characters to the google.co.uk url to see US results - just can't remember what video I saw it in. Any ideas? Thanks a lot
International SEO | | trbaldwin0