Blocking domestic Google's in Robots.txt
-
Hey,
I want to block Google.co.uk from crawling a site but want Google.de to crawl it.
I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers?
any ideas?
Thanks
B
-
Thanks Guys for all of the help.
I think we will just implement cross domain GeoIP redirects to ensure users get the right location and currency.
Cheers
-
Are you having the issue of your .de pages ranking in .co.uk instead of your .co.uk pages?
If that's the case then I'd look towards usage of HREFLANG both on-page and in the xml sitemaps. That is going to provided Googlebot with a better view of the country-language targeting for the site.
-
Hi, country specific search engine spiders cannot be blocked using robots.txt file or any other method. However, you can block certain IP ranges pertaining to certain countries.
Best regards,
Devanur Rafi
-
Hi Gareth,
I don't think this is going to work as every crawler by Google is run by the same useragent: Googlebot. What you could do but what I really wouldn't recommend to do is generating the robots.txt automatically. Check if the IP address of the user is in another country and then Disallow them. It probably won't work as the crawler from let's say Germany could also be used for the UK.
Also the specific data for a countries search engine gets collected first and then gets looked at to see what they need users to serve, not the other way around that content gets acquired for a specific country.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to check Robots.txt File
I want to know that how we are going to check that the Robots.txt File of the website is working properly. Kindly elaborate the mechanism for it please.
International SEO | | seobac1 -
Google Search Console "International Targeting" is reporting errors that are not present on my site
We are currently handling search for a global brand www.example.com/ which has presence in many countries worldwide. To help Google understand that there is an alternate version of the website available in another language, we have used hreflang tags. These hreflang tags are implemented only via the XML sitemap across all geo-locations. Under the “Search Analytics -> International Targeting” section, in Google Search Console, for the Malaysian website (www.example.com/my/), there are a number of “no-return tags (sitemaps)” errors arising. For example, for India as a geo-location, there is one ‘en-IN’ – no return tags (sitemaps) errors listed. The error is listed below: Originating URL - www.example.com/my/xyz/ Alternate URL - www.example.com/in/xyz/ When the XML sitemap for the URL – www.example.com/in/ was checked for the hreflang tags, it was noticed that the implementation of hreflang tags for the URL – www.example.com/in/xyz/ was perfectly fine and it was providing a return tag to the URL – www.example.com/my/xyz/. After the code level verification, it was identified that the implementation of hreflang tags was perfectly fine via the XML sitemap. Even though at the code level it was verified that the implementation is fine, the error still persists in Google Search Console. Kindly suggest a solution to this situation, and also advise the effects of these errors on search engine performance
International SEO | | Starcom_Search0 -
Trying To Use Parent Company's Content In Another Country
Hi, I'm hoping someone can help me out here but this is what I am dealing with: Say John Smith Companies sells Widgets across the United States. They have also formed a company called "Widgets of Canada" in an effort to sell their Blue Widgets only in Canada and I am in responsible for their website. Recently, John Smith Companies completely redesigned their website and it now has a really slick look and is loaded with great widgets content. I would like to take their site and re-purpose it for use in Canada. However, I am concerned about duplicate content. I would be converting all the widget specifications from imperial to metric units, changing the title and description elements and also using a much different folders/ paths. Is this enough to avoid any issues with similar page content? Is there anything I can do with hreflang? Thanks
International SEO | | DohenyDrones0 -
Hotel Multi country targeting + Google Local + TLDs => "MesSEO"
Hi guys, I own a guesthouse which is facing a messy structural problem in its own web presence: Portuguese: www.residencia-aeminium-coimbra.com.pt English: www.residencia-aeminium-coimbra.com Spanish: www.residencia-aeminium-coimbra.com/espanol Looking for success in the long term, a few years ago, we decided to host 3 TLDs: 1 for global international english, 1 for local portuguese and 1 for the main foreign market Spain (we already redirected it to a subfolder in the .com in the meanwhile). We tried to promote each one of these in their targeted markets - mainly the .com got back-links links and authority. With time, google local appeared and changed the face of google accommodation rankings. Google local are now the top results for most searches, even outside the country, and I came to the conclusion that the TLD assets were now a problem. For instance, I can only add 1 domain to google local..what language should this be? It is the same for most social media presence..How should international markets be targeted in a local page? Since TLDs do not seem to be an advantage right now, I am thinking we should be moving to a single domain and use a folder structure so we can use it everywhere. Questions: 1. In terms of structure (TLD or Folder) and multi-lingual targeting what is the current best practice for hotels that show in local results but promote internationally? 2. What language should I point google local results to? And our Facebook page, etc? 3. If I move things around to a folder structure, what domain should I use? the .com is in english and has the most authority and links according to opensiteexplorer the .com.pt is in portuguese, our local language and our main market (but only 35% share) should I create a new domain ".pt"? 4. I don't think that geo-targeting the languages is worth it in this case, what do you think? Kind Regards
International SEO | | retsimister
Ricardo Madeira
Residencia Aeminium Coimbra0 -
Google Webmaster Tools - Geographic target - Time till change
One of my clients is targeting UK customers. They have UK IP address. I then noticed that they had their Geographic target set to "Target users in: United States". I advised them to change this. Anyone know: How this could have been set, the client says they didn't do it? How long will it take for Google to see this has been changed? Thanks in advance. Justin
International SEO | | GrouchyKids0 -
How do I successfully verify my site for Baidu's webmaster tools?
Instructions for verifying a website via file validation for Baidu's webmaster tools are pretty vague. Does anyone know if the process is the same as Google Webmaster Tools where the verification string must appear in the URL and in the content of the file? Also, does it truly have to be verified within 2.6 hours? Appreciate any feedback from people who have successfully verified their site.
International SEO | | sigmaaldrich0 -
How do you see Google results specific to location?
We run a Canadian website and are interested in seeing what SERPs look like from specific postal codes. Is there any way to manipulate Google to think our IP address comes from another location? Thanks!
International SEO | | ClaytonKendall0 -
What countries does Google crawl from? Is it only US or do they crawl from Europe and Asia, etc.?
Where does Google crawl the web from? Is it in the US only, or do they do it from a European base too? The reason for asking is for GeoIP redirection. For example, if a website is using GeoIP redirection to redirect all US traffic to a .com site and all EU traffic to a .co.uk site, will Google ever see the .co.uk site?
International SEO | | Envoke-Marketing2