The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Forum
    2. Categories
    3. SEO Tactics
    4. Technical SEO
    5. Moz crawler is not able to crawl my website

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Moz crawler is not able to crawl my website

    Technical SEO
    moz crawler
    2 2 2.8k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • JasonTorney
      JasonTorney last edited by

      Hi, i need help regarding Moz Can't Crawl Your Site i also share screenshot that Moz was unable to crawl your site on Mar 26, 2022. Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.
      my robts.txt also ok i checked it
      Here is my website https://whiskcreative.com.au
      just check it please as soon as possibe

      JCN-SBWD 1 Reply Last reply Reply Quote 0
      • JCN-SBWD
        JCN-SBWD @JasonTorney last edited by

        @jasontorney Hi, I had the same problem after moving several of my websites to a Virtual Private Server that had enhanced security features.

        One of these features was specifically to stop the Moz bot from crawling websites, and the hosting engineers advised they had done this because it was particularly aggressive in nature.

        With my VPS control panel I have found the switch that allows me to disable bot blocking, and I occasionally do this if I'm grading a page with Moz, but advice from hosting support was to otherwise leave it active to protect the websites from attached (which means I don't get feedback from Moz crawls).

        If you check with your hosting company you may find that they have a similar bot blocker configured for security purposes.

        1 Reply Last reply Reply Quote 0
        • 1 / 1
        • First post
          Last post

        Got a burning SEO question?

        Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


        Start my free trial


        Explore more categories

        • Moz Tools

          Chat with the community about the Moz tools.

          Getting Started
          Moz Pro
          Moz Local
          Moz Bar
          API
          What's New

        • SEO Tactics

          Discuss the SEO process with fellow marketers

          Content Development
          Competitive Research
          Keyword Research
          Link Building
          On-Page Optimization
          Technical SEO
          Reporting & Analytics
          Intermediate & Advanced SEO
          Image & Video Optimization
          International SEO
          Local SEO

        • Community

          Discuss industry events, jobs, and news!

          Moz Blog
          Moz News
          Industry News
          Jobs and Opportunities
          SEO Learn Center
          Whiteboard Friday

        • Digital Marketing

          Chat about tactics outside of SEO

          Affiliate Marketing
          Branding
          Conversion Rate Optimization
          Web Design
          Paid Search Marketing
          Social Media

        • Research & Trends

          Dive into research and trends in the search industry.

          SERP Trends
          Search Behavior
          Algorithm Updates
          White Hat / Black Hat SEO
          Other SEO Tools

        • Support

          Connect on product support and feature requests.

          Product Support
          Feature Requests
          Participate in User Research

        • See all categories

        • The Moz.com bot is overloading my server
          paulavervo
          paulavervo
          0
          8
          873

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        • Digital Marketers
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy

        Looks like your connection to Moz was lost, please wait while we try to reconnect.