The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Forum
    2. Categories
    3. Moz Tools
    4. Getting Started
    5. Moz not able to crawl our site - any advice?

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Moz not able to crawl our site - any advice?

    Getting Started
    5 5 1.2k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • MyFamilClubLtd
      MyFamilClubLtd last edited by

      When I try and crawl our site through Moz it gives this message:

      Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster.

      I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt.

      Anyone got any advice on where I could go to get this sorted?

      1 Reply Last reply Reply Quote 1
      • Riverdance
        Riverdance last edited by

        Hello, have the same issue and MOZ getting "Bummer no data found" for domain https://wherebuycoin.com/ . The SEMRush and Ahrefs see this site, but MOZ not, even after 2 years. What can be wrong? Where to read why MOZ not crawling my site?

        Thanks!

        1 Reply Last reply Reply Quote 0
        • dgghyjuyu
          dgghyjuyu last edited by

          Check your website's robots.txt file to ensure that it's not blocking Moz's user-agent or specific directories/pages that you want to be crawled. Update the robots.txt file if necessary to allow Moz to crawl your site.

          1 Reply Last reply Reply Quote 0
          • Salience_Search_Marketing
            Salience_Search_Marketing last edited by

            Hey,

            We crawled the site just fine with Screaming Frog. Perhaps this is a Moz issue?

            Looks like they are well aware of it though.

            Thanks!

            1 Reply Last reply Reply Quote 0
            • A Former User
              A Former User last edited by

              Hi,

              I'm sorry to hear about this! Would you be able to write into help@moz.com as this will allows us to investigate into this via the most convenient method.

              Looking forward to hearing from you,

              Eli

              1 Reply Last reply Reply Quote 1
              • 1 / 1
              • First post
                Last post

              Got a burning SEO question?

              Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


              Start my free trial


              Explore more categories

              • Moz Tools

                Chat with the community about the Moz tools.

                Getting Started
                Moz Pro
                Moz Local
                Moz Bar
                API
                What's New

              • SEO Tactics

                Discuss the SEO process with fellow marketers

                Content Development
                Competitive Research
                Keyword Research
                Link Building
                On-Page Optimization
                Technical SEO
                Reporting & Analytics
                Intermediate & Advanced SEO
                Image & Video Optimization
                International SEO
                Local SEO

              • Community

                Discuss industry events, jobs, and news!

                Moz Blog
                Moz News
                Industry News
                Jobs and Opportunities
                SEO Learn Center
                Whiteboard Friday

              • Digital Marketing

                Chat about tactics outside of SEO

                Affiliate Marketing
                Branding
                Conversion Rate Optimization
                Web Design
                Paid Search Marketing
                Social Media

              • Research & Trends

                Dive into research and trends in the search industry.

                SERP Trends
                Search Behavior
                Algorithm Updates
                White Hat / Black Hat SEO
                Other SEO Tools

              • Support

                Connect on product support and feature requests.

                Product Support
                Feature Requests
                Participate in User Research

              • See all categories

              • Moz can't crawl my site.
                bluerinsevintage
                bluerinsevintage
                0
                2
                1.0k

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              • Digital Marketers
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy

              Looks like your connection to Moz was lost, please wait while we try to reconnect.