Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. Moz Tools
    3. Moz Pro
    4. WEbsite cannot be crawled

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    WEbsite cannot be crawled

    Moz Pro
    2
    2
    961
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • threecounties
      threecounties last edited by

      I have received the following message from MOZ on a few of our websites now

      Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.

      I have spoken with our webmaster and they have advised the below:

      The Robots.txt file is definitely there on all pages and Google is able to crawl for these files. Moz however is having some difficulty with finding the files when there is a particular redirect in place.

      For example, the page currently redirects from threecounties.co.uk/ to https://www.threecounties.co.uk/ and when this happens, the Moz crawler cannot find the robots.txt on the first URL and this generates the reports you have been receiving. From what I understand, this is a flaw with the Moz software and not something that we could fix form our end.

      _Going forward, something we could do is remove these rewrite rules to www., but these are useful redirects and removing them would likely have SEO implications. _

      Has anyone else had this issue and is there anything we can do to rectify, or should we leave as is?

      1 Reply Last reply Reply Quote 0
      • Roman-Delcarmen
        Roman-Delcarmen last edited by

        Ok, I made a quick test of your robot.txt file and looks fine,
        https://www.threecounties.co.uk/robots.txt

        Then I made a test https://httpstatus.io/ to check the status code
        of your robot.txt file and show me 200 status code (So it's fine)

        Also, you need to make sure that your robot.txt file is accessible for the Rogerbot (Moz crawler)
        This day the hosting providers have become very strict with third-party crawlers
        This includes Moz, Majestic SEO, Semrush and Ahrefs.

        Here you can find all the possible sources of the problem and recommended solutions
        https://moz.com/help/guides/moz-pro-overview/site-crawl/unable-to-crawl

        Regards

        1 Reply Last reply Reply Quote 1
        • 1 / 1
        • First post
          Last post

        Got a burning SEO question?

        Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


        Start my free trial


        Browse Questions

        Explore more categories

        • Moz Tools

          Chat with the community about the Moz tools.

        • SEO Tactics

          Discuss the SEO process with fellow marketers

        • Community

          Discuss industry events, jobs, and news!

        • Digital Marketing

          Chat about tactics outside of SEO

        • Research & Trends

          Dive into research and trends in the search industry.

        • Support

          Connect on product support and feature requests.

        • See all categories

        Related Questions

        • ChunkyChuck

          How can i increase DA of my website?

          domain authority

          Hi, Why my website evolvepakistan DA is not increasing it is old site even i have bought a fiverr gig of someone he was offering that he will show my site DA 70, he send me pic of 70DA, on his side 70DA of my site but when i check on it show 0 DA.
          Whats the problem?

          Moz Pro | | ChunkyChuck
          0
        • tunguyen110894

          How test my website?

          Help me test my website? My website: United Airlines

          Moz Pro | | tunguyen110894
          2
        • Dogara

          Ive been using moz for just a minute now , i used it to check my website and find quite a number of errors , unfortunately i use a wordpress website and even with the tips , is till dont know how to fix the issues.

          ive seen quite a number of errors on my website hipmack.co  a wordpress website and i dont know how to begin clearing the index errors or any others for that matter , can you help me please? ghg-1.jpg

          Moz Pro | | Dogara
          0
        • Ben-R

          Why Only Our Homepage Can Be Crawled Showing a Redirect Message as the Meta Title

          Hello Everyone, So recently when we checked our domain using a Moz Crawl Test and Screaming Frog only the homepage comes up and the meta title says “You are being redirected to…”. We have several pages that used to come up and when submitting them to GSC no issues come up. The robots.txt file looks fine as well. We thought this might be ‘server’ related but it’s a little out of our field of expertise so we thought we would find out if anyone has any experience with this (ideas of reasons, how to check…etc.) or any potential suggestions. Any extra insight would be really appreciated. Please let us know if there is anything we could provide further details for that might. Looking forward to hearing from all of you! Thanks in advance. Best,

          Moz Pro | | Ben-R
          0
        • MEllsworth

          Potential spam websites with high DA linking back to us

          Hey everybody, I'm going through all my sites and disavowing crap links. However, I'm having trouble distinguishing which high DA sites to disavow. What would you do? For example:
          https://moz.com/researchtools/ose/spam-analysis?site=busca.starmedia.com&target=domain&source=subdomain&page=1&sort=spam_score and https://moz.com/researchtools/ose/spam-analysis?site=cc879fe.activerain.com&target=domain&source=subdomain&page=1&sort=spam_score They both have tons of backlinks - both good and crap. The first has a DA of 72 and a Moz spam score of 4/17 and the second has a DA of 86 and a Moz spam score of 9/17

          Moz Pro | | MEllsworth
          1
        • RyanShahed

          A tool to tell a websites estimated traffic

          I am new to Moz (as a member), so I am not sure if Moz has a tool that I need. I don't want this post to be about self promotion, so I will keep it short. Our business helps increase conversions and sales for online businesses. Our ideal prospects belongs to some key categories of businesses like ecommerce, saas etc. However, I would like to know the estimated volume of traffic for a website before approaching them and introducing our service. So if there was a tool I could use to estimate the volume of visitors a specific website receives on average a day or month, it would be hugely beneficial.Obviously, these are prospective clients, so we do not have access to their system or their analytics. I just want to get an estimate. So for example, if I entered the domain abc.com into the system, I would hope it could tell me, that abc.com gets an average of 900 unique visitors a day. I don't need too much detail like geographic locations etc, but it would be a bonus having that additional information. I also don't mind paying for a tool that's quality. So it doesn't have to be free.

          Moz Pro | | RyanShahed
          0
        • PlanesTrainsAutos

          Adding new categories to my website and getting them ranked

          Hi, I currently run a website and have added a couple of new categories lately. The website is Planes Trains Automobiles Co UK. For example I want to add "Planes for sale" as a new category. What is the best way to go about this? Will each aircraft added add content therefore ranking? I don't want to approach an SEO company yet as the service would be free for the time being? Thanks

          Moz Pro | | PlanesTrainsAutos
          0
        • oumma

          How to force SeoMoz to re-crawl my website?

          Hi, I have done a lot of changes on my website to comply with SeoMoz advices, now I would like to see if I have better feedback from the tool, how can I force it to re-crawl a specific campaign? (waiting another week is too long :-))

          Moz Pro | | oumma
          0

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy

        Looks like your connection to Moz was lost, please wait while we try to reconnect.