Blocking other engines in robots.txt
-
If your primary target of business is not in China is their any benefit to blocking Chinese search robots in robots.txt?
-
I don't see any benefit to blocking search engines with robots.txt with the exception of Bing or Google as necessary.
Robots.txt is strictly a suggestion to those crawlers who care enough to respect your wishes.
The only benefit it can offer is IF a crawler chooses to respect your wishes, then your site will have a bit less traffic volume during the crawl. The reality is any specific crawler from one of the many random companies is going to visit so infrequently it wont make any noticeable difference.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Role of Robots.txt and Search Console parameters settings
Hi, wondering if anyone can point me to resources or explain the difference between these two. If a site has url parameters disallowed in Robots.txt is it redundant to edit settings in Search Console parameters to anything other than "Let Googlebot Decide"?
Technical SEO | | LivDetrick0 -
Bloking pages in roborts.txt that are under a redirected subdomain
Hi Everyone, I have a lot of Marketo landing pages that I don't want to show in SERP. Adding the noindex meta tag for each page will be too much, I have thousands of pages. Blocking it in roborts.txt could have been an option, BUT, the subdomain homepage is redirected to my main domain (with a 302) so I may confuse search engines ( should they follow the redirect or should they block) marketo.mydomain.com is redirected to www.mydomain.com disallow: / (I think this will be confusing with the redirect) I don't have folders, all pages are under the subdomain, so I can't block folders in Robots.txt also Would anyone had this scenario or any suggestions? I appreciate your thoughts here. Thank you Rachel
Technical SEO | | RaquelSaiz0 -
One robots.txt file for multiple sites?
I have 2 sites hosted with Blue Host and was told to put the robots.txt in the root folder and just use the one robots.txt for both sites. Is this right? It seems wrong. I want to block certain things on one site. Thanks for the help, Rena
Technical SEO | | renalynd270 -
Google webmaster… Zopim Live chat blocking the resources
I have noticed that Zopim Live chat blocking the resources (URL’s). What are the reasons? Now I have fetched blocked urls again for indexing purpose but still I want to know the reason. Any help from experts??? yYJJ8oM.jpg
Technical SEO | | udistm0 -
Do I have a robots.txt problem?
I have the little yellow exclamation point under my robots.txt fetch as you can see here- http://imgur.com/wuWdtvO This version shows no errors or warnings- http://imgur.com/uqbmbug Under the tester I can currently see the latest version. This site hasn't changed URLs recently, and we haven't made any changes to the robots.txt file for two years. This problem just started in the last month. Should I worry?
Technical SEO | | EcommerceSite0 -
Block or remove pages using a robots.txt
I want to use robots.txt to prevent googlebot access the specific folder on the server, Please tell me if the syntax below is correct User-Agent: Googlebot Disallow: /folder/ I want to use robots.txt to prevent google image index the images of my website , Please tell me if the syntax below is correct User-agent: Googlebot-Image Disallow: /
Technical SEO | | semer0 -
Block url with dynamic text in
I've just ran a report and I have a lot of duplicate page titles, most of which seem to be the review page, I use Magento and my normal url would be something like blah-blahtext.html but the review url is something like blah-blahtext/reviews/category/categoryname So I want to block the /reviews url bit as no one ever leaves reviews and it's not something I will be using in the future. Also I have a dynamic navigation which creates urls that look like product-name.html?size=2&colour=14 these are also creating duplicate urls, anyway to fix this? While I'm asking, anyone any tips for Magento?
Technical SEO | | Beermonster0 -
Advice on display this content on my page for search engines
Hi, my website http://www.in2town.co.uk/Holiday-News is about bringing travel and holiday news to our readers of our lifestyle magazine but i am having problems at the moment with the layout. What i mean by this is, i have written content on the page as an introduction so google knows what this section of the site is about but to be honest it looks rubbish with having the introduction there and i would like to know if i am doing the right thing by having the content there for google to know what my site is about. I have tried taking it away and noticed i dropped in the rankings and when i have put it back up i go up in the rankings, can anyone please give me some advice over this issue
Technical SEO | | ClaireH-1848860