undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. Moz Tools
  3. Moz Pro
  4. Robots.txt blocking Moz

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Robots.txt blocking Moz

Moz Pro
2
6
1.7k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • ChristinaRadisic
    ChristinaRadisic last edited by Aug 3, 2017, 5:19 AM

    Moz are reporting the robots.txt file is blocking them from crawling one of our websites.

    But as far as we can see this file is exactly the same as the robots.txt files on other websites that Moz is crawling without problems.

    We have never come up against this before, even with this site.

    Our stats show Rogerbot attempting to crawl our site, but it receives a 404 error.

    Can anyone enlighten us to the problem please?

    http://www.wychwoodflooring.com

    -Christina

    1 Reply Last reply Reply Quote 0
    • ChristinaRadisic
      ChristinaRadisic last edited by Aug 4, 2017, 6:44 AM Aug 4, 2017, 6:44 AM

      Hi Nigel

      Neither, they use server side filtering.Regards- David

      1 Reply Last reply Reply Quote 0
      • Nigel_Carr
        Nigel_Carr @ChristinaRadisic last edited by Aug 4, 2017, 6:30 AM Aug 4, 2017, 6:30 AM

        Hi David

        That's great news!

        As a matter of interest, where did they block it? as it's not in the Robots.txt - was in in htaccess.txt?

        Regards

        Nigel

        1 Reply Last reply Reply Quote 0
        • ChristinaRadisic
          ChristinaRadisic @Nigel_Carr last edited by Aug 4, 2017, 6:04 AM Aug 4, 2017, 6:04 AM

          Nigel,Thanks for the reply, the cgi-bin folder is never used by any of my sites but I put this in just as a matter of course, the folder would normally contain old cgi scripts so would not usually affect the crawling of a robot in any case.The reason for the problem turns out that our host had blocked rogerbot along with several other malicious bots, they have now lifted this block and the site is able to be crawled.- David

          Nigel_Carr 1 Reply Last reply Aug 4, 2017, 6:30 AM Reply Quote 0
          • Nigel_Carr
            Nigel_Carr last edited by Aug 3, 2017, 2:16 PM Aug 3, 2017, 2:16 PM

            Hi Christina

            I don't know how your site is set up but I can see that for some reason you are blocking access to the cgi-bin

            If that directory contains files that execute php or other permissions then that may well be your problem. It's the only directory you are blocking and since I haven't seen other Robots.tx blocking it, then I would hazard a guess that this is the root of your problem.

            Robots.txt

            User-agent: *
            Disallow: /cgi-bin/
            
            Sitemap: http://www.wychwoodflooring.com/sitemap.xml
            

            Regards

            Nigel

            ChristinaRadisic 1 Reply Last reply Aug 4, 2017, 6:04 AM Reply Quote 0
            • ChristinaRadisic
              ChristinaRadisic last edited by Aug 3, 2017, 6:58 AM Aug 3, 2017, 6:58 AM

              Our hosting provider has banned Rogerbot as they see it as problematic!!!!

              They are a great hosting provider so this is going to be a difficult one.

              1 Reply Last reply Reply Quote 0
              • 1 / 1
              1 out of 6
              • First post
                1/6
                Last post

              Got a burning SEO question?

              Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


              Start my free trial


              Browse Questions

              Explore more categories

              • Moz Tools

                Chat with the community about the Moz tools.

              • SEO Tactics

                Discuss the SEO process with fellow marketers

              • Community

                Discuss industry events, jobs, and news!

              • Digital Marketing

                Chat about tactics outside of SEO

              • Research & Trends

                Dive into research and trends in the search industry.

              • Support

                Connect on product support and feature requests.

              • See all categories

              Related Questions

              • gallowaywebteam

                Unsolved Does Moz Pro include Moz Local

                moz local moz pro

                My client has bought about six Moz Local accounts and are pleased with results. We have not yet used your Moz Pro program. The client might be interested in switching to the Moz Pro if those Moz Local accounts can be included into it. Please let me know as soon as possible. Thanks!

                Moz Pro | Jan 18, 2024, 3:02 PM | gallowaywebteam
                0
              • NaomiAdivare

                Ahrefs vs Moz

                Hi! I noticed the Moz DA en the Ahrefs DA are very different. Where https://www.123opzeggen.nl/ has a DA of 10 at MOZ, the DA at Ahrefs is 26. Where does this big difference come from? Do you measure in different ways? I hope you can answer this question for me. Thank you in advance!

                Moz Pro | Nov 17, 2023, 7:48 AM | NaomiAdivare
                2
              • Sequelmed

                MoZ vs SEMRush - Keyword Difficulty and Keyword With Low Competition.

                Hi, My question is very focussed regarding these 2 tools. Is it correct to understand that MoZ tells keyword difficulty and not which keyword is easy to compete. And SEMRush tells which keyword is easy to compete but does not tell which keywords are difficult to compete? I mean one (Moz)  misses the one part and the other (SEMRush) misses the other part. Hope some will enlighten me to the point. Best

                Moz Pro | Nov 24, 2017, 5:43 PM | Sequelmed
                0
              • shankar333

                What is Linking C-Blocks

                Currently i am using MOZ pro tool under moz analyticls >> Moz Competitive Link Metrics >> history having a graph "Linking C-Blocks" Please help me understanding Linking C-Blocks, what is, How to build, how to define ...

                Moz Pro | Oct 30, 2015, 3:28 PM | shankar333
                5
              • Blacktie

                Block Moz (or any other robot) from crawling pages with specific URLs

                Hello! Moz reports that my site has around 380 duplicate page content. Most of them come from dynamic generated URLs that have some specific parameters. I have sorted this out for Google in webmaster tools (the new Google Search Console) by blocking the pages with these parameters. However, Moz is still reporting the same amount of duplicate content pages and, to stop it, I know I must use robots.txt. The trick is that, I don't want to block every page, but just the pages with specific parameters. I want to do this because among these 380 pages there are some other pages with no parameters (or different parameters) that I need to take care of. Basically, I need to clean this list to be able to use the feature properly in the future. I have read through Moz forums and found a few topics related to this, but there is no clear answer on how to block only pages with specific URLs. Therefore, I have done my research and come up with these lines for robots.txt: User-agent: dotbot
                Disallow: /*numberOfStars=0 User-agent: rogerbot
                Disallow: /*numberOfStars=0 My questions: 1. Are the above lines correct and would block Moz (dotbot and rogerbot) from crawling only pages that have numberOfStars=0 parameter in their URLs, leaving other pages intact? 2. Do I need to have an empty line between the two groups? (I mean between "Disallow: /*numberOfStars=0" and "User-agent: rogerbot")? (or does it even matter?) I think this would help many people as there is no clear answer on how to block crawling only pages with specific URLs. Moreover, this should be valid for any robot out there. Thank you for your help!

                Moz Pro | Jul 21, 2015, 11:43 AM | Blacktie
                0
              • rcman

                Why is MOZ and Google search Volume so different?

                A search term in MOZ shows the monthly search volume to be 49K. In Google, the same term shows the search volume at only 1300 monthly searches. Which do I trust? Thanks, Don

                Moz Pro | Jan 12, 2015, 11:52 AM | rcman
                0
              • LesleyPaone

                How you can manipulate your MOZ DA

                I have become frustrated at MOZ in the last few months, none of my backlinks have made it into the index. Old back links. Long story short, I figured out the issue and I figured out how anyone can manipulate their DA. I wrote a blog post about it here, http://blog.dh42.com/manipulate-moz/

                Moz Pro | Jan 3, 2014, 11:54 PM | LesleyPaone
                1
              • cschwartzel

                Can you add a user to your MOz account or to a single campaign?

                Can you add a user to your MOz account or to a single campaign?

                Moz Pro | Oct 5, 2015, 11:55 AM | cschwartzel
                1

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy

              Looks like your connection to Moz was lost, please wait while we try to reconnect.