undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. Feature Requests
  3. Moz crawler is not able to crawl my website

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Moz crawler is not able to crawl my website

Feature Requests
4
5
1.4k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • Harini.M
    Harini.M last edited by Sep 15, 2017, 2:33 AM

    Hello All,

    I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. "

    We changed the robots.txt file and checked it . but still the issue is not resolved.

    URL : https://www.khadination.shop/robots.txt

    Do let me know what went wrong and wjhat needs to be done.

    Any suggestion is appreciated.

    Thank you.

    1 Reply Last reply Reply Quote 0
    • jocameron
      jocameron Staff last edited by Sep 22, 2017, 8:11 AM Sep 22, 2017, 8:11 AM

      Hi Harini,

      Jo from the Moz help team here.

      I've had a look at your site and it looks like there is something server side that is blocking our bot.

      When I try to cURL your site from our internal tool I'm getting a 302 to http://127.0.0.1

      https://screencast.com/t/J3hhDTCM

      I'm also seeing this message in this third party tool.

      "The robots.txt file does not exist on this domain (302 redirect to http://127.0.0.1)"

      All this points to something server side that is initiating a 302 redirect for our bot. While your site looks fine in the browser, our bot simply can't get through.

      I would recommend reaching out to your host or web developer to see if they can check how your server is treating rogerbot/1.2

      You can also ask them to check the server logs to see how your server is responding to rogerbot/1.2

      You'll also want to make sure you are not blocking AWS (Amazon Web Services).

      Best of luck!

      Jo

      1 Reply Last reply Reply Quote 0
      • Harini.M
        Harini.M @andy.bigbangthemes last edited by Sep 18, 2017, 4:05 AM Sep 18, 2017, 4:05 AM

        Thank you Andy 🙂 . But, the problem is MOZ crawler was unable to crawl the website even though the line " Allow: / " was present in the robots.txt.

        User-agent: *
        Allow: /
        Disallow: /wp-admin/
        Disallow: /wp-includes/
        Disallow: /?color=
        Disallow: /
        ?manufacturer=
        Disallow: /
        ?filter_material-fabric=
        Disallow: /
        ?filter_color=
        Disallow: /
        ?query_type_color=
        Disallow: /
        ?filter_size=
        Disallow: /
        ?taxonomy=
        Disallow: /?view_mode=
        Disallow: /
        ?query_type_material-fabric=
        Disallow: /?orderby=
        Disallow: /
        ?source_id=
        Disallow: /?source_tax=
        Disallow: /
        ?shop-2__trashed?
        Allow: /wp-admin/admin-ajax.php

        Sitemap: https://www.khadination.shop/sitemap.xml

        this was the previous version of robots.txt that were been used ....

        1 Reply Last reply Reply Quote 0
        • SMRBathrooms
          SMRBathrooms last edited by Sep 15, 2017, 12:52 PM Sep 15, 2017, 12:52 PM

          Hi - As Andy has said, you're not allowing Moz to crawl the site.

          Read up on Rogerbot here: https://moz.com/help/guides/moz-procedures/what-is-rogerbot

          1 Reply Last reply Reply Quote 1
          • andy.bigbangthemes
            andy.bigbangthemes last edited by Sep 15, 2017, 6:52 AM Sep 15, 2017, 6:52 AM

            Hi there,

            You forgot the most important thing. You're disallowing a lot of things but not allowing access in the first place.

            Allow: /

            add this on line 2 of your robots.txt file.

            Good luck

            Harini.M 1 Reply Last reply Sep 18, 2017, 4:05 AM Reply Quote 1
            • 1 / 1
            1 out of 5
            • First post
              1/5
              Last post

            Got a burning SEO question?

            Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


            Start my free trial


            Browse Questions

            Explore more categories

            • Moz Tools

              Chat with the community about the Moz tools.

            • SEO Tactics

              Discuss the SEO process with fellow marketers

            • Community

              Discuss industry events, jobs, and news!

            • Digital Marketing

              Chat about tactics outside of SEO

            • Research & Trends

              Dive into research and trends in the search industry.

            • Support

              Connect on product support and feature requests.

            • See all categories

            Related Questions

            • datenwerk

              Unsolved Matomo Data Integration

              matomo

              As GA4 is not easily usable for all clients in the EU, it would be great if you could offer a Matomo integration. Matomo has recently been more and more popular and is used by a lot of companies, that are required to adhere to EU standards in data privacy.

              Feature Requests | Nov 17, 2023, 11:32 AM | datenwerk
              0
            • darkwebvampire

              Why my site's DA and Pa Not Show Accurate By MOZ Extension.

              my site has some minor problem which about DA and PA that DA and PA do not show accurate some time my site is yts subtitle, please tell me something about it.

              Feature Requests | Mar 25, 2024, 11:59 AM | darkwebvampire
              0
            • RaquelSaiz

              Keyword keyword rank checker

              Where is the Moz tool/page where you can check rankings of keywords for any site or page? Thanks Rachel

              Feature Requests | May 4, 2018, 8:39 PM | RaquelSaiz
              0
            • midcenturybliss

              One of my Moz Campaigns was accidentally deleted. Any tips on getting it back?

              I had a campaign setup with all of my keywords meticulously labeled and branded. Then by mistake, it was deleted. Is there any way I can restore/get the data back for an accidentally deleted campaign?

              Feature Requests | Aug 20, 2017, 6:36 PM | midcenturybliss
              0
            • brettmandoes

              Is there an API to download keyword data?

              Howdy fellow Mozzers, We're tracking a client's keyword rankings in one of our campaigns and want to export that data on a regular basis to display in Tableau. Tableau let's us quickly match up different data sources and create visuals with the data. It's useful for marrying different data sources that don't normally talk to each other. We know we can pull keyword tracking data manually, but we're hoping there's an API or scheduler that we can use to download the data automatically. Automating the process would be super helpful. Thank you!

              Feature Requests | May 31, 2017, 5:28 PM | brettmandoes
              0
            • Cenktukel

              Does seo moz toolbar work in safari? Can I avoid using Chrome or firefox?

              I am using a Mac and use Safari for web.  I am not very familiar with Google Chrome or Firefox. I started my trial with SEOMoz Pro and want to use it (loved it) but do not want to adapt myself to Google Chrome or Firefox - thus my hesitation. What limitations do I have with SEOMoz Pro in safari - e.g. Moz add-on in the toolbar? What would I be missing if I continue using Safari? I would much appreciate an honest and complete answer so that I can decide whether to use Google Chrome of Firefox. Thank you

              Feature Requests | Apr 7, 2017, 6:48 PM | Cenktukel
              0
            • NewWindow

              External Link Detection Very Slow... why?

              Hello. I'm new to MOZ and I'm now comparing it with ahrefs. I can't afford to have an account with all SEO tools that are out there, but I have to start somewhere. I really like MOZ a lot. However, there is one major issue - it updates backlinks every 3 weeks, or what? I registered to ahrefs trial today just to make sure I don't regret in the future that I bought the first service that I tried (MOZ), and I was amazed that ahrefs showed all my website's backlinks straight away. MOZ is saying that next scheduled update for Externals Links is going to be on Sept 14.... emm sorry? My trial will run out by then. It now shows exactly 0. Pricing is nearly the same and in Today's fast paced world ahrefs seems to tick the ever so important backlinks quality box. So here is my question. Dear MOZ, are you going to speed up the backlink data delivery? Are you working on it, or you think it's just ahrefs's gimmick? If you can promise faster speeds, I'll be happy to stay subscribed with MOZ and wait because I like the UX a lot. However, at the end of the day speed is important and if my client requests the latest report, I don't want to tell him that he needs to wait a month for it. Many thanks for reading and I hope to hear back from someone.

              Feature Requests | Sep 10, 2016, 4:08 PM | NewWindow
              1
            • Whittie

              Do MOZ have an affiliate platform?

              Do MOZ have an affiliate platform? If so, it's hiding from me, would somebody care to point me the right way? Shall I assume this is the page / they MOZ don't have a direct affiliate program? https://moz.com/partners  Many Thanks

              Feature Requests | Jun 22, 2015, 2:54 PM | Whittie
              0

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy

            Looks like your connection to Moz was lost, please wait while we try to reconnect.