What countries does Google crawl from? Is it only US or do they crawl from Europe and Asia, etc.?
-
Where does Google crawl the web from? Is it in the US only, or do they do it from a European base too? The reason for asking is for GeoIP redirection. For example, if a website is using GeoIP redirection to redirect all US traffic to a .com site and all EU traffic to a .co.uk site, will Google ever see the .co.uk site?
-
Hi Keith,
In my experience Google mainly crawls from the US.
You're quite right, GeoIP redirection can cause major issues with indexation - as if you're redirecting everything from the US to the .com googlebot can't see the .co.uk site.
As such, I'm not a fan. Rather than implementing a hard redirect I prefer Amazon's approach. If you visit the amazon.com site from a UK IP you get a javascript overlay that invites you to visit the .co.uk version of the site instead - they let the user decide which site to view rather than actually redirecting them.
This is a nice solution as it ensures that the search bots can crawl both versions of the site, and rankings aren't endangered.
I hope this helps,
Hannah
-
Keith, I am having the same issue and I agree with you. The fact that Google has data centers in Europe does not necessarily mean the algos are indexing from there. I also want to set Europe and US GeoIP redirection. It would be great to get Mozers opinions on this. Hopefully this post gets freshly reviewed
-
Interesting question - I'd quite like to know what happens here too.
Matt Cutts recently posted a video on cloaking (http://youtu.be/QHtnfOgp65Q) saying that as long as you don't do anything 'special' for Googlebot you're OK, but presumably if you are redirecting IP's based on location and you don't want to prevent Googlebot form accessing your site then effectively you have to do something 'special' for Googlebot (e.g. you're doing one thing for everyone else and a different thing for Googlebot).
-
Hi,
The data center locations is interesting, but it isn't what I was looking for. I need to know whether Google crawls the web from any IP other than US IPs.
To clear up the second question, let me be more specific:
Let's say Google is crawling a .co.uk site from a US IP address. The site is using GEO IP redirection to redirect all US traffic to the .com site. Therefore, when Google attempts to crawl the .co.uk from the US IP address site it will be redirected to the .com site, never seeing the .co.uk site. Can anyone confirm that this is what happens?
-
I found an article from 2008 that shows Google data centre locations: http://bit.ly/mONhf9
Your other question is a bit confusing. Why Google wouldn't see the UK site?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Which Google does a .ie website get shown in?
Hi all, I am working on a .ie website and I was under the understanding that if you have a regional domain, like .ie you will limited to being shown in a search engine like google.ie When I go to International Targeting in Google SEarch Console it says the site is associated with: Ireland Am I limiting my ability to rank well in worldwide Google searches with this domain and if so, how can I counter this? Many thanks.
International SEO | | Bee1590 -
Web Site Migration - Time to Google indexing
Soon we will do a website migration .com.br to .com/pt-br. Wi will do this migration when we have with lower traffic. Trying to follow Google Guidelines, applying the 301 redirect, sitemap etc... I would like to know, how long time the Google generally will use to transfering the relevance of .com.br to .com/pt-br/ using redirect 301?
International SEO | | mobic0 -
Google Analytics & Webmaster Tools Filtering
I've just set up a client who uses internationalization on Google Webmaster tools and Google Analytics. For easier management they opted to use subfolders rather than subdomains or cctlds. So I set up a Google Analytic Property, with one Unfiltered profile, and another 3 profiles filtered per language. With the main language English being exclude anything starting /fr/ /de/ as it resides on root. The filters seem to work fine; however after linking this to the Google Webmaster Account to be able to access Search Engine Optimization I do not seem to get any language filtered data. I was wondering if someone had any idea or possible solution to this problem. As I would expect to at least have the Landing Pages if not exactly the keywords filtered by the same criteria that the rest of the data is being filtered. I know there's also an option to create a separate webmaster tools account, however this way I still cannot filter just the English; and I cannot link it to all the separate profiles.
International SEO | | jonmifsud0 -
Google suggesting a translation
Hi everybody, I notice since some months that Google when used for german language results proposes a translation next to the listing of one of my websites. When searching for english results (hl=en) it does not propose a translation! My website is clearly in german (given as target in GWMT and by meta tag). Other pages on the same domain are not treated this way by Google. No translation is proposed for all subpages of this website. Obviously, Google considers the homepage of this website english instead of german. Any fix for that? It is a *.org Wolfgang
International SEO | | wgr_strategic0 -
Ranking issues for UK vs US spelling - advice please
Hi guys, I'm reaching out here for what may seem to be a very simple and obvious issue, but not something I can find a good answer for. We have a .com site hosted in Germany that serves our worldwide audience. The site is in English, but our business language is British (UK) English. This means that we rank very well for (e.g.) optimisation software but optimization software is nowhere to be found. The cause of this to me seems obvious; a robot reading those two phrases sees two distinct words. Nonetheless, having seen discussions of a similar nature around the use of plurals in keywords, it would seem to me that Google should have this sort of thing covered. Am I right or wrong here? If I'm wrong, then what are my options? I really don't want to have to make a copy of the entire site; apart from the additional effort involved in content upkeep I see this path fraught with duplicate content issues. Any help is very much appreciated, thanks.
International SEO | | StevenHowe0 -
Will Google punish me cuz my websites content are almost the same?
If I have almost the same contents for my three e-commerce websites, say A.com,B.uk,C.ca. They're promoted in US, GB, Canada which are all English speaking. Will my site be punished because they're almost the same to Google?
International SEO | | SquallPersun0 -
Country specific landing pages
I have a client who wants to put a re-direct on his landing pages based on the visitors IP address. The landing page will be a sub domain relevant to the country their IP is located in. I am a little concerned this will effect the SEO. Appreciate any advice. Dylan 🙂
International SEO | | gomyseo0 -
Targeting specific Geographic areas. Use 1 large.Com or several smaller country specific TLDs?
Hi, I have a small number of exact match domains, both country specific TLDs and also the Generic TLD dot com and dot net. They are: ExactMatch**.Com**
International SEO | | Hurf
ExactMatch**.Net** ExactMatch**.Co.Uk**
ExactMatch**.Ca**
ExactMatch**.Co.Nz**
ExactMatch**.Co.Za** We have already successfully launched our UK site using the exact match .co.uk and this is currently number 2 in the UK SERPS for the Google, Yahoo and Bing. They are/will be niche specific classified ad sites, which are Geographically targeted by country (to Engish speakers in the main) and each region is likely to have a minumum of 2,000 unique listings submitted over the course of a year of so. My question (FINALLY) is this: Am I better to build one large global site (will grow to approx. 12,000 listings) using EXACTMATCH.Com with .com - targeting US users and then geo-targeted sub directories (ExactMatch.Com/Nz etc) - each sub dir targeted to the matching geographic area in webmaster tools, or use the ccTLDs and host each site in the country with perhaps (each site growing to approx 2,000 listings) I could use the ccTLDs just for marketing/branding onlyand redirect these to the specific sub directory of the .com site? I am aware that there is one main ccTLD that I cannot get .Com.Au (as I am not a resident of Australia - and it is already in use.) so I was wondering if the single site with .Com/AU/ etc might help me better target that country? If I use each ccTLD as separate sites I suppose I could use the largely redundant .net to target Australia? Your thoughts and advice would be most welcome. Thanks! An additional bit of intormation (or two) the .com is circa 2004. The product advertised is a reasonably bulky (perhaps 6kgs boxed) physical product and therefore the seller is unlikely to want to ship globally - will this make them shy away from a global site - even one divided into global sub sections? FYI Seller can specify in their listing Will Ship To ....... I would be open to looking at using the front page of the .Com site as a page which visitors select the country they wish to buy/sell on. (IF it is the general consensus that it is better to create one large site.) Consider also please how the end user is likely to percieve the benefits to them of one LARGE SITE versus TARGETED SITE - I know the .Com would be divided into geographic sub directories, but I am not sure if they won't see an additinal benefit to the ccTLD - Does this add a degree of reassurance and relevance that a .com/ccTLD cannot provide? I suppose I am biased by the fact that ebay use ccTLDs? Thanks again - and please forgive my tone which may suggest I am playing devil's advocate here. I am very torn on this issue.0