Blocking Competitive Analysis Tools in robots.txt.... Worth it?
-
I've been considering blocking third party crawlers for a while – specifically those crawling my website for the sake of competitive analysis, such as SEMrush and Ahrefs. I'm familiar with how to do so, but when researching the question I found practically no one asking the same question. The guides I've found on what to put in your robots.txt make no mention of whether to block competitive analysis crawlers. Which makes me wonder whether this is a good idea after all.
My chief concern here is rival sites going after the same search terms we target – one of our competitors in particular has an uncanny way of going after the same searches we are. I know blocking crawlers won't prevent competitors from watching our content, but it will make it slightly harder for them. Is there any major drawback I'm missing? Any big reason not to go ahead and block SEO analysis crawlers?
-
Hi David,
Well, if you use a tool that can change the name of the bot that comes after a site, then you can't do anything about it. However, I do know that SEMrush is causing problems for some hosts and the likes of Heart Internet, they block access to the SEMrush bot on all of their servers.
To be honest with you, it is quite difficult to block everything and time consuming and unless you have some secret formula to keywords that are magically drawing in more traffic, then I wouldn't worry about it too much.
Take Accuranker. I use this for tracking and part of setting up, is it will find new phrases that I perhaps haven't been using or looking at before. It doesn't get these from either SEMrush or Ahrefs. All of this information is available from many sources.
I would honestly spend the time making your pages as good as they can be. Look at your cometitors. Are they winning? How are they doing it? Fight back against them rather trying to hide yourself from view.
-Andy
-
Issa is correct and to be honest, he's pointed out what I see as a pretty major problem - not being able to see these metrics (particularly ahrefs) really isn't ideal for analysis and/or troubleshooting.
Also, it may very well be that the competitor that keeps targeting the same terms as you is doing so via other means. There are countless tools available to all of us that allow for analysis of strong and relevant terms so your site may have little to do with their decisions.
Beyond this, one glance at your page titles will often be enough to figure out what you're targeting
-
Hi David,
This is completely your choice. Your website's SEO and overall ranking wont be affected by block SEMRush for example.
The only drawback I can think of is that you wont be able to use this tool for your own domain research, which can be helpful if you're looking for ranking stats, or click share from search engines. I would hate to not be able to research my client's websites regularly, as this is the kind of intelligence I use to a) understand Google's and Users behaviour to find my website. b) discover technical and SEO issues with my websites.
So although you can go a head and do it, I would say you should always be ready to unblock them when needed. Unfortunately SEM Rush does not update their database so regularly, so you wouldn't be able to predict when you should unblock their robot for information.
I hope I could help
Issa
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best competitor analysis tool comparing my links with competitor links?
Hello All, Can anyone tell me best competitor analysis tool comparing my links with competitor links? Tool which define quality of links either low quality or spammy links etc? Thanks!
Competitive Research | | devdan0 -
Currently Number 2 - Worth Shooting for 1?
Hello SEO Community, So I was able to rise to #2 in a highly competitive search term after a guest blog post that I wrote really took off (wish I could recreate that for my other terms, haha) Anyway, I steadily rose from 30ish to number 2 and have obviously seen more traffic, which is awesome and I'm happy. But still, in terms of the amount of traffic that google estimates the keyword gets, and the amount I'm getting at #2, it seems like #1 is getting the VAST majority (I know this is how most markets work). Judging from my link profile compared to #1, it would be extremely difficult to dethrone them. So what do y'all think? Is it worth spending time and resources shooting for the glory and traffic of #1 and potentially never getting there - or focusing my efforts on my middling keywords and getting those on the first page as well?
Competitive Research | | CleanEdisonInc0 -
Competitor research: No data / results displaying on Keyword tools, Aexa
Hi there! I'm trying to research a few competitors using various Keyword tools (SEM Rush, Compete, Keyword Spy -- even Alexa for high level insight). While the bulk of the competitors generate expected results through these tools (a smattering of their top organic and paid search keywords, some traffic estimations through Alexa), ONE of these competitors lists "No results" across all categories and all tools: http://www.bgstar.ca Despite this, we know that they invest heavily in search -- and my SEM Rush toolbar indicates that they have a Google PR of 5 (though I recognize that that should be taken with a grain of salt). So I'm stumped! Has anyone encountered this before? Is there something structural that they might be doing, that's blocking not only Google-based platforms, but Alexa too? Thanks for your help!
Competitive Research | | MACJ0 -
Tool to compare which links competition has but i don't?
Is there a tool that I can use that will show me a list of my competition's links that I don't have?
Competitive Research | | ballhogjoni0 -
Good tools to compare site speed on different sites?
Google tells me that my site is slower than 81% of the internet.Now this is a very large news website, so I would like to compare this against other very large news websites. Are there any tools that will be able to tell me the speed of a list of websites? I am looking for something a little more detailed than 'this site loaded in 1 second'. Also, do any of you fellow SEOs know of any good tools to analyse what is causing your site to slow down? Kind regards
Competitive Research | | MirandaP0 -
What's the best way to make sure a link is worth getting?
I know what tools to use and I use all of SEOmoz's tools daily. PA, DA, MR, & MT are all things I take into account, but sometimes all you have to do is look at a site and you can tell it's not worth it. I'll analyze the page's backlink's and everything in between. Are there any tricks out there that can help the decision making process? I'm tired of trying to get links on sites that clearly are not worth it, but all of their stats say otherwise. So do you stick with stats, or is it a judgement call? I'm particularly curious about determining the amount of possible traffic from a link. Any ideas would be greatly appreciated!
Competitive Research | | MichaelWeisbaum0 -
Best Traffic Estimation Tool
Hello, What is the best system/tool to estimate traffic to a site, something more accurate than Alexa rank. This is of course assuming that one does not have access to the site's Google Analytics data, Thanks, Carl
Competitive Research | | CarlLarson0 -
How do you perform competitive research for SEO?
What metrics tell you the most when you're looking at your competitors across the search landscape? PageRank/MozRank Inbound links Keyword rankings Alexa/QuantCast/etc. Pages indexed Something else entirely? What numbers speak volumes to you when you want to get an idea of how you benchmark against your competitors? And how do you communicate these results?
Competitive Research | | jcolman2