undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Competitive Research
  4. Blocking Competitive Analysis Tools in robots.txt.... Worth it?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Blocking Competitive Analysis Tools in robots.txt.... Worth it?

Competitive Research
4
4
3.6k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • davidwaring
    davidwaring Subscriber last edited by May 12, 2016, 6:16 PM

    I've been considering blocking third party crawlers for a while – specifically those crawling my website for the sake of competitive analysis, such as SEMrush and Ahrefs. I'm familiar with how to do so, but when researching the question I found practically no one asking the same question. The guides I've found on what to put in your robots.txt make no mention of whether to block competitive analysis crawlers. Which makes me wonder whether this is a good idea after all.

    My chief concern here is rival sites going after the same search terms we target – one of our competitors in particular has an uncanny way of going after the same searches we are. I know blocking crawlers won't prevent competitors from watching our content, but it will make it slightly harder for them. Is there any major drawback I'm missing? Any big reason not to go ahead and block SEO analysis crawlers?

    1 Reply Last reply Reply Quote 0
    • Andy.Drinkwater
      Andy.Drinkwater last edited by May 13, 2016, 5:39 AM May 13, 2016, 5:39 AM

      Hi David,

      Well, if you use a tool that can change the name of the bot that comes after a site, then you can't do anything about it. However, I do know that SEMrush is causing problems for some hosts and the likes of Heart Internet, they block access to the SEMrush bot on all of their servers.

      To be honest with you, it is quite difficult to block everything and time consuming and unless you have some secret formula to keywords that are magically drawing in more traffic, then I wouldn't worry about it too much.

      Take Accuranker. I use this for tracking and part of setting up, is it will find new phrases that I perhaps haven't been using or looking at before. It doesn't get these from either SEMrush or Ahrefs. All of this information is available from many sources.

      I would honestly spend the time making your pages as good as they can be. Look at your cometitors. Are they winning? How are they doing it? Fight back against them rather trying to hide yourself from view.

      -Andy

      1 Reply Last reply Reply Quote 1
      • ChrisAshton
        ChrisAshton last edited by May 12, 2016, 10:15 PM May 12, 2016, 10:15 PM

        Issa is correct and to be honest, he's pointed out what I see as a pretty major problem - not being able to see these metrics (particularly ahrefs) really isn't ideal for analysis and/or troubleshooting.

        Also, it may very well be that the competitor that keeps targeting the same terms as you is doing so via other means. There are countless tools available to all of us that allow for analysis of strong and relevant terms so your site may have little to do with their decisions.

        Beyond this, one glance at your page titles will often be enough to figure out what you're targeting 🙂

        1 Reply Last reply Reply Quote 0
        • iQi
          iQi last edited by May 12, 2016, 7:32 PM May 12, 2016, 7:32 PM

          Hi David,

          This is completely your choice. Your website's SEO and overall ranking wont be affected by block SEMRush for example.

          The only drawback I can think of is that you wont be able to use this tool for your own domain research, which can be helpful if you're looking for ranking stats, or click share from search engines. I would hate to not be able to research my client's websites regularly, as this is the kind of intelligence I use to a) understand Google's and Users behaviour to find my website. b) discover technical and SEO issues with my websites.

          So although you can go a head and do it, I would say you should always be ready to unblock them when needed. Unfortunately SEM Rush does not update their database so regularly, so you wouldn't be able to predict when you should unblock their robot for information.

          I hope I could help

          Issa

          1 Reply Last reply Reply Quote 0
          • 1 / 1
          1 out of 4
          • First post
            1/4
            Last post

          Got a burning SEO question?

          Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


          Start my free trial


          Browse Questions

          Explore more categories

          • Moz Tools

            Chat with the community about the Moz tools.

          • SEO Tactics

            Discuss the SEO process with fellow marketers

          • Community

            Discuss industry events, jobs, and news!

          • Digital Marketing

            Chat about tactics outside of SEO

          • Research & Trends

            Dive into research and trends in the search industry.

          • Support

            Connect on product support and feature requests.

          • See all categories

          Related Questions

          • Cricket93

            Search Intent and Relevance

            competitor search intent

            Hello SEO gurus 🤓 I’m looking for the most efficient ways to analyze the search intent and relevance of competitors who are ranking for the keywords we’re targeting. While I know Google excels at assessing search intent and relevance, I’m interested in learning how we can evaluate these factors as metrics for our competitors. The goal is to understand their strategy better and find ways to outrank them. Do you have any tools or methodologies that you recommend for assessing competitor content to determine its alignment with search intent and user needs/relevance? I’d love to hear your thoughts and suggestions on this!

            Competitive Research | Nov 1, 2024, 8:37 PM | Cricket93
            1
          • grayloon

            What is the relationship between inbound links and linking domains?

            inbound link linking root domains

            In doing competitor research, there's a specific competitor that has fewer linking domains, but dramatically more inbound links. How do these two relate to each other? I do have a higher domain authority and more internal followed links with them having more internal nofollow and external (both follow and nofollow) links. I'm trying to understand how they can have 3.7k linking domains and 4.8 million inbound links while I have 4.1k linking domains and only 601k inbound links. What really defines an inbound link and what's there relationship between inbound links and linking domains?

            Competitive Research | Sep 7, 2022, 7:40 AM | grayloon
            0
          • thanhdung0906

            Interal linking

            tipbongda

            Re: I'm stuck by internal linking. What structure should a football website follow? Silo or Topic Cluster?
            I need advice on my website. My website: https://tipbongda247.net/
            I hope there are answers!
            Thanks

            Competitive Research | Jul 5, 2024, 5:41 PM | thanhdung0906
            0
          • Constantin2021

            Rank online sex shop

            I have been trying for some time to take the online store to the front page, but it is very difficult to go up in google searches for the keyword ( sex shop ).
            Can anyone help me with some tips. Thank you.
            https://www.voluptas.ro/

            Competitive Research | Feb 17, 2021, 11:10 AM | Constantin2021
            0
          • igniterman75

            How do you select which keywords to push in SEO?

            Hi Guys Selecting the right keywords that a website can realistically rank for is a key to gain top rankings relatively quick.  I am just curious to hear how you guys do it  (the methodology) when selecting which keywords to push? I mean you need to check the competition for each keyword as well so how to check this quickly to see what we realistically can rank for? Cheers John

            Competitive Research | Apr 19, 2024, 6:52 AM | igniterman75
            1
          • tonyflorespsa

            Why do some sites perform great but show no data on the tool bar?

            Hey everyone, Been a paid member for 3 days and this is my first time posting 🙂 I have a quick question about competitive analysis with the MozBar.  I do not want to point out the exact website (unless that is something cool to do on this forum), but there is a site ranking for a single keyword in the local 7 pack in slot one, which is also slot one for the SERP entirely.  When I go to look at the page in more detail, the MozBar is showing no DA, PA, anything at all, and the Open Site Explorer is showing no links internal or external. I know that the page has to have at least some of these elements populated in order to be ranking for a term this competitive, so why is the bar not showing anything?  The bar works GREAT on other sites below this one. Thanks for any help!

            Competitive Research | May 27, 2013, 3:01 AM | tonyflorespsa
            0
          • monster99

            Free tools to find country of origin of backlinks/urls

            Hey are there any free tools out there which can allow me to insert a large list of urls, and it determines the country of origin of the domain. I know the paid version of majestic does, but i was wondering if theres any free tools? Cheers, Chris

            Competitive Research | Dec 11, 2012, 7:00 PM | monster99
            0
          • technota

            My competition is using 1px images and links without anchor text. Good or bad idea?

            I've been scoping out some of my competition and I noticed the number 1 ranking site is doing a lot of "interesting" stuff.  I saw a bunch of 1px blank images that they must be using for the alt tag keywords.  I also noticed several links to internal pages with different keywords without anchor text so they don't actually appear on the page.  Is this considered black hat?  Should I be doing something similar?  It feels like the work I'm doing to try to rank #1 with a clean/professional looking site are wasted when the #1 sites are doing things like this.

            Competitive Research | Nov 13, 2011, 3:09 AM | technota
            0

          Get started with Moz Pro!

          Unlock the power of advanced SEO tools and data-driven insights.

          Start my free trial
          Products
          • Moz Pro
          • Moz Local
          • Moz API
          • Moz Data
          • STAT
          • Product Updates
          Moz Solutions
          • SMB Solutions
          • Agency Solutions
          • Enterprise Solutions
          Free SEO Tools
          • Domain Authority Checker
          • Link Explorer
          • Keyword Explorer
          • Competitive Research
          • Brand Authority Checker
          • Local Citation Checker
          • MozBar Extension
          • MozCast
          Resources
          • Blog
          • SEO Learning Center
          • Help Hub
          • Beginner's Guide to SEO
          • How-to Guides
          • Moz Academy
          • API Docs
          About Moz
          • About
          • Team
          • Careers
          • Contact
          Why Moz
          • Case Studies
          • Testimonials
          Get Involved
          • Become an Affiliate
          • MozCon
          • Webinars
          • Practical Marketer Series
          • MozPod
          Connect with us

          Contact the Help team

          Join our newsletter
          Moz logo
          © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
          • Accessibility
          • Terms of Use
          • Privacy

          Looks like your connection to Moz was lost, please wait while we try to reconnect.