Blocking Competitive Analysis Tools in robots.txt.... Worth it?
-
I've been considering blocking third party crawlers for a while – specifically those crawling my website for the sake of competitive analysis, such as SEMrush and Ahrefs. I'm familiar with how to do so, but when researching the question I found practically no one asking the same question. The guides I've found on what to put in your robots.txt make no mention of whether to block competitive analysis crawlers. Which makes me wonder whether this is a good idea after all.
My chief concern here is rival sites going after the same search terms we target – one of our competitors in particular has an uncanny way of going after the same searches we are. I know blocking crawlers won't prevent competitors from watching our content, but it will make it slightly harder for them. Is there any major drawback I'm missing? Any big reason not to go ahead and block SEO analysis crawlers?
-
Hi David,
Well, if you use a tool that can change the name of the bot that comes after a site, then you can't do anything about it. However, I do know that SEMrush is causing problems for some hosts and the likes of Heart Internet, they block access to the SEMrush bot on all of their servers.
To be honest with you, it is quite difficult to block everything and time consuming and unless you have some secret formula to keywords that are magically drawing in more traffic, then I wouldn't worry about it too much.
Take Accuranker. I use this for tracking and part of setting up, is it will find new phrases that I perhaps haven't been using or looking at before. It doesn't get these from either SEMrush or Ahrefs. All of this information is available from many sources.
I would honestly spend the time making your pages as good as they can be. Look at your cometitors. Are they winning? How are they doing it? Fight back against them rather trying to hide yourself from view.
-Andy
-
Issa is correct and to be honest, he's pointed out what I see as a pretty major problem - not being able to see these metrics (particularly ahrefs) really isn't ideal for analysis and/or troubleshooting.
Also, it may very well be that the competitor that keeps targeting the same terms as you is doing so via other means. There are countless tools available to all of us that allow for analysis of strong and relevant terms so your site may have little to do with their decisions.
Beyond this, one glance at your page titles will often be enough to figure out what you're targeting
-
Hi David,
This is completely your choice. Your website's SEO and overall ranking wont be affected by block SEMRush for example.
The only drawback I can think of is that you wont be able to use this tool for your own domain research, which can be helpful if you're looking for ranking stats, or click share from search engines. I would hate to not be able to research my client's websites regularly, as this is the kind of intelligence I use to a) understand Google's and Users behaviour to find my website. b) discover technical and SEO issues with my websites.
So although you can go a head and do it, I would say you should always be ready to unblock them when needed. Unfortunately SEM Rush does not update their database so regularly, so you wouldn't be able to predict when you should unblock their robot for information.
I hope I could help
Issa
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What tools can give reliable competitor traffic data?
Could you give details of reliable tools that can measure and provide competitors' traffic data? Google Adplanner used to provide such information until June of last year, and while it's numbers didn't match Google Analytics data, the numbers were reasonably close. The metrics we are most interested in are visits, page views and unique visitors. We have looked at sites such as Compete -- but feel it is far too inaccurate/ skewed with US data only; also, would prefer to not rely on Alexa traffic stats. Thanks in advance.
Competitive Research | | ontarget-media0 -
Competitive Domain Analysis
What is the best way to improve this in reference to my competition? My site(s) are formatted well, returning no errors or warnings. I have removed my keyword meta-tag and tested my sites against the keywords that we want to appear under. Now I need to close the gap on the competition.
Competitive Research | | bobbabuoy0 -
Looking for a recommendation for a site ranking comparison tool.
Hi, It seems that SEOMOZ provides only link ranking comparison with competators. I'm not conviced that this is valuable for my purposes as the comeption often has few links tool, probably the nature of the business.. First, is it worth trying to compare keyword ranking with competators? I think it is, but I'm not that experienced with SEO, especially with Wordpress/Buddpress sites. Second, any recommenations for a tool that will make this easier? Larry
Competitive Research | | tishimself0 -
Www.seoprofiler.com - anyone knows the service? worth the investment?
A little help just to check if the service worth the money spend.
Competitive Research | | Naghirniac0 -
Looking For Specific Type Of Tool - Maybe You Know It?
I'm looking for tool where I can insert a keyword phrase and have it give me a list of websites with a percentage of the traffic they receive: example: keyword is "knee brace" result: Amazon 35.16% next site 24% next site 13.54% etc Does anyone currently use a website or know of a tool that can provide this information. Free sites would also be doubly appreciated! Thank you Mozzers.
Competitive Research | | dmedirect_sbcglobal.net0 -
Keyword difficulty tool - bit confused here!!
Hi, Got a question about the keyword difficulty tool, I'm new to seomoz so might be a silly question but here goes. Q: It takes the top 10 results from Google under a certain search query, the results displayed gives a rough breakdown of what power the page on the search results has along with the root domain. It looks like it rewards the best out of the bunch for each section with a tick, so 1 tick per column for the best performer. Now I would have thought if one of the websites in the top 10 had all 4 ticks it would rank at the top of the list but this is not the case. What else would come into play to make others which have no ticks rank higher then one which has all. Sorry if this sounds really confusing.
Competitive Research | | activitysuper0 -
Question regarding back link analysis and anchor text
Hello, I am looking at my competitors back link analysis and comparing a range of link based metrics from the top 10 SERPS. I am then putting this data into excel and comparing our back link profiles. When looking at the anchor text distribution i am not sure whether to look at exact match anchor or phrase match anchor. For example, one of the companies I am looking at holds positions 3 and 4 in the SERPS. Looking at their linking profile I can see that only 1.7% of their links use the exact match anchor 'widget'. Looking at their phrase match anchor is an entirely different story, 93.5% of anchor links contain 'widget' somewhere. i.e. 'cheap widgets', 'widget sale', 'buy widgets at www.examplewidget.co.uk' etc. Obviously their exact match and phrase match anchor distribution tell a completely different story. THIS IS TRUE FOR MANY OF THE TOP 10 SERPS. Therefore, should I be looking at phrase match anchors instead of exact? Side note: would people recommend targeting anchors with 'brandname widget' based on predictions of Google giving weighted anchor more weight. Robert.
Competitive Research | | 87ROB0 -
How do you perform competitive research for SEO?
What metrics tell you the most when you're looking at your competitors across the search landscape? PageRank/MozRank Inbound links Keyword rankings Alexa/QuantCast/etc. Pages indexed Something else entirely? What numbers speak volumes to you when you want to get an idea of how you benchmark against your competitors? And how do you communicate these results?
Competitive Research | | jcolman2