undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    What is your Brand Authority?
    Moz

    What is your Brand Authority?

    Check yours now
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • SEO Q&A

      Insights & discussions from an SEO community of 500,000+.

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Technical SEO
  4. Crawl rate dropped to zero

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Crawl rate dropped to zero

Technical SEO
2
13
2.3k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • pok3rplay3r
    pok3rplay3r last edited by Mar 22, 2016, 5:05 PM

    Hello, I recently moved my site in godaddy from cpanel to managed wordpress. I bought this transfer directly from GoDaddy customer service. in this process they accidentally changed my domain from www to non www. I changed it back after the migration, but as a result of this sites craw rate from search console fell to zero and has not risen at all since then.

    In addition to this website does not display any other errors, i can ask google manually fetch my pages and it works as before, only the crawl rates seems to be dropped permanently. GoDaddy customer service also claims that do not see any errors but I think, however, that in some way they caused this during the migration when the url changed since the timing match perfectly.  also when they accidentally removed the www, crawl rate of my sites  non www version got up but fell back to zero when I changed it back to www version. Now the crawl rate of both www and non www version is zero. How do I get it to rise again? Customer service also said that the problem may be related to ftp-data of search console?  But they were not able to help any more than .Would someone from here be able to help me with this in anyway please?

    1 Reply Last reply Reply Quote 0
    • pok3rplay3r
      pok3rplay3r last edited by Apr 2, 2016, 8:41 AM Apr 2, 2016, 8:41 AM

      Hello, asnwers to the questions bolded:

      • At this rate, how long would it take Google to crawl all of your pages, (maybe it feels 10-15 is fast enough)? Over 50 days, i still cannot believe that it would be just a coincidence that crawl rate dropped so suddenly only because google suddenly thinks that my page should not be crawled that often. After all, amount of new content, quality of new links and all the other factors are much better all the time on my site, and before the drop, crawl rate increased steadily. It has to be some technical issue?
      • Has the average response time increased? If so, maybe Google feels it's overloading the server & backing off. No, it has actually went down a little bit (not much though)
      1 Reply Last reply Reply Quote 0
      • 4RS_John
        4RS_John Subscriber @pok3rplay3r last edited by Apr 1, 2016, 10:56 AM Apr 1, 2016, 10:56 AM

        Interesting. I have 2 more thoughts:

        • At this rate, how long would it take Google to crawl all of your pages, (maybe it feels 10-15 is fast enough)?
        • Has the average response time increased? If so, maybe Google feels it's overloading the server & backing off.
        1 Reply Last reply Reply Quote 0
        • pok3rplay3r
          pok3rplay3r last edited by Apr 1, 2016, 4:57 AM Apr 1, 2016, 4:57 AM

          Crawl rate still is extremely slow, average 10-15 per day except when i sent pages to be manually crawled, then it crawls those page. Before the drop the crawl rate was never under 200 per day and it was usually over 1000. anything more I can do? It seems to have no effect my rankings or anything else as l can see, but I still would like this be fixed. It has be something to do with the fact that i changed my hosting to godaddy managed wordpress hosting. but they have no clue about what could cause this. robot.txt file change seemed to have no effect or very minimum effect

          4RS_John 1 Reply Last reply Apr 1, 2016, 10:56 AM Reply Quote 0
          • 4RS_John
            4RS_John Subscriber @pok3rplay3r last edited by Mar 23, 2016, 4:05 PM Mar 23, 2016, 4:05 PM

            Not that I'm aware of, unfortunately. Patience is an important skill when dealing with Google 😉

            1 Reply Last reply Reply Quote 0
            • pok3rplay3r
              pok3rplay3r last edited by Mar 23, 2016, 3:15 PM Mar 23, 2016, 3:15 PM

              Thanks! I will try that. I see that search console shows crawl rates with few days delay, is there somewhere i could check if it works instantly?

              4RS_John 1 Reply Last reply Mar 23, 2016, 4:05 PM Reply Quote 0
              • 4RS_John
                4RS_John Subscriber @pok3rplay3r last edited by Mar 23, 2016, 10:31 AM Mar 23, 2016, 10:31 AM

                I thought of one other possibility: Your sitemap.xml is probably auto-generated, so this shouldn't be a problem, but check to make sure that the URLs in the sitemap.xml have the www.

                Other than that I'm out of ideas - I would wait a few days to see what happens, but maybe someone else with more experience watching Google will have seen this before. If it does resolve, I'd like to know what worked.

                1 Reply Last reply Reply Quote 1
                • 4RS_John
                  4RS_John Subscriber @pok3rplay3r last edited by Mar 23, 2016, 10:29 AM Mar 23, 2016, 10:29 AM

                  I'm not convinced that robots.txt is causing your problem, but it can't hurt to change it back. In fact, while looking for instructions on how to change it I came across this blog post by Joost de Valk, (aka Yoast), that pretty much says you should remove everything that's currently in your robots.txt - and his arguments are right for everything:

                  • Blocking wp-content/plugins will stop Google from loading JS and/or CSS resources that it might need to render the page properly.
                  • Blocking wp-admin is redundant, because the wp-admin if it's linked it can still be found, and important pages already have an X-Robots HTTP header that says not to index them.

                  If you're using Yoast SEO, here are instructions on how to change the robots.txt file.

                  1 Reply Last reply Reply Quote 1
                  • pok3rplay3r
                    pok3rplay3r last edited by Mar 23, 2016, 9:17 AM Mar 23, 2016, 9:17 AM

                    Hi, one more thing. Are you 100% sure tht robot.txt file hs nothing to do with this? It changed at the sime time when the problems started to occur. It used to be :

                    User-agent: *
                    Disallow: /wp-admin/
                    Allow: /wp-admin/admin-ajax.php

                    But now it is :

                    User-agent: *
                    Crawl-delay: 1
                    Disallow: /wp-content/plugins/
                    Disallow: /wp-admin/

                    At the sime time "blocked resources" notifications started to occur in search console.

                    Blocked Resources >  Rendering without certain resources can impair the indexing of your web pages. Learn more.Status: 3/19/16152 Pages with blocked resources

                    This has to have something to do with it right?

                    4RS_John 1 Reply Last reply Mar 23, 2016, 10:29 AM Reply Quote 0
                    • pok3rplay3r
                      pok3rplay3r last edited by Mar 23, 2016, 6:56 AM Mar 23, 2016, 6:56 AM

                      Thank you for your answer, my answers bolded here below

                      • Do you see any crawl errors in the Google Search Console?  **Nothing new after the crawl rate dropped, just some old soft 404 errors and old not found errors. **
                      • If you search for your site on Google, what do you see, (does your snippet look normal)? Yes everything looks perfectly normal, just like before when the crawl rate dropped
                      • How many pages does Google say it has indexed? Is it possible it's indexed everything and is taking a break, (does it even do that?) I dont thin this is possible, since the cralw rate dropped lmost instantly from average 400 to zero after the site migration.

                      One theory is: When you moved to the non-www version of the site, Google started getting 301s redirecting it from www to non-www, and now that you've gone back to www it's getting 301s redirecting it from from non-www to www, so it's got a circular redirect. If this is the problem, how should i start to get it fixed?

                      Here's what I would do to try to kick-start indexing, if you haven't already:

                      • Make sure you have the "Preferred Domain" set to the www version of your site in_ both the www and non-www versions of your site_ in Google Search Console. Yes that is how it has been all the time
                      • In the Search Console for the www-version of your site, re-submit your sitemap. Done
                      • In the Search Console for the www-version of your site, do a Fetch as Google on your homepage, and maybe a couple of other pages, and when the Fetch is done use the option to submit those pages for indexing, (there's a monthly limit on how much of this you can do). I have done this many times since i noticed the problem, fetch as google works normally without any issues

                      Is there anything more i can do? If i want hire someone to fix this, is there any recommendations? I am not a tech guy so this is quite difficult task for me

                      4RS_John 1 Reply Last reply Mar 23, 2016, 10:31 AM Reply Quote 0
                      • 4RS_John
                        4RS_John Subscriber last edited by Mar 22, 2016, 11:22 PM Mar 22, 2016, 11:22 PM

                        I don't know why this is happening, but this is what I would check:

                        • Do you see any crawl errors in the Google Search Console?
                        • If you search for your site on Google, what do you see, (does your snippet look normal)?
                        • How many pages does Google say it has indexed? Is it possible it's indexed everything and is taking a break, (does it even do that?)

                        One theory is: When you moved to the non-www version of the site, Google started getting 301s redirecting it from www to non-www, and now that you've gone back to www it's getting 301s redirecting it from from non-www to www, so it's got a circular redirect.

                        Here's what I would do to try to kick-start indexing, if you haven't already:

                        • Make sure you have the "Preferred Domain" set to the www version of your site in both the www and non-www versions of your site in Google Search Console.
                        • In the Search Console for the www-version of your site, re-submit your sitemap.
                        • In the Search Console for the www-version of your site, do a Fetch as Google on your homepage, and maybe a couple of other pages, and when the Fetch is done use the option to submit those pages for indexing, (there's a monthly limit on how much of this you can do).

                        Good luck!

                        1 Reply Last reply Reply Quote 1
                        • 4RS_John
                          4RS_John Subscriber @pok3rplay3r last edited by Mar 22, 2016, 11:06 PM Mar 22, 2016, 11:06 PM

                          That's not so horrible - it just says not to crawl the plugins directory or the admin, and to delay a second between requests. You probably don't want your plugins or admin directories being indexed, and according to this old forum post Google ignores the crawl-delay directive, so the robots.txt isn't the problem.

                          1 Reply Last reply Reply Quote 1
                          • pok3rplay3r
                            pok3rplay3r last edited by Mar 22, 2016, 8:16 PM Mar 22, 2016, 8:16 PM

                            Hi, my robot.txt file looks like this:

                            User-agent: *
                            Crawl-delay: 1
                            Disallow: /wp-content/plugins/
                            Disallow: /wp-admin/
                            
                            This is not how it suppose to look like, right? could this cause the problem?
                            
                            4RS_John 1 Reply Last reply Mar 22, 2016, 11:06 PM Reply Quote 0
                            • 1 / 1
                            1 out of 13
                            • First post
                              1/13
                              Last post

                            Got a burning SEO question?

                            Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                            Start my free trial


                            Browse Questions

                            Explore more categories

                            • Moz Tools

                              Chat with the community about the Moz tools.

                            • SEO Tactics

                              Discuss the SEO process with fellow marketers

                            • Community

                              Discuss industry events, jobs, and news!

                            • Digital Marketing

                              Chat about tactics outside of SEO

                            • Research & Trends

                              Dive into research and trends in the search industry.

                            • Support

                              Connect on product support and feature requests.

                            • See all categories

                            Related Questions

                            • LisaBabblebird

                              Domain Authority dropped 10 points and I don't know why

                              domain authority optimization

                              In my latest site crawl, the domain authority dropped 10 points for no apparent reason. There have been no changes to the site. The only change I have made this month is to block referral spam to the site. My competitors' DAs have stayed the same too. website name: https://knowledgefront.co.uk/ Any ideas?

                              Technical SEO | Aug 9, 2022, 8:24 AM | LisaBabblebird
                              0
                            • TimDickey

                              Any crawl issues with TLS 1.3?

                              Not a techie here...maybe this is to be expected, but ever since one of my client sites has switched to TLS 1.3, I've had a couple of crawl issues and other hiccups. First, I noticed that I can't use HTTPSTATUS.io any more...it renders an error message for URLs on the site in question. I wrote to their support desk and they said they haven't updated to 1.3 yet. Bummer, because I loved httpstatus.io's functionality, esp. getting bulk reports. Also, my Moz campaign crawls were failing. We are setting up a robots.txt directive to allow rogerbot (and the other bot), and will see if that works. These fails are consistent with the date we switched to 1.3, and some testing confirmed it. Anyone else seeing these types of issues, and can suggest any workarounds, solves, hacks to make my life easier? (including an alternative to httpstatus.io...I have and use screaming frog...not as slick, I'm afraid!) Do you think there was a configuration error with the client's TLS 1.3 upgrade, or maybe they're using a problematic/older version of 1.3?? Thanks -

                              Technical SEO | May 14, 2021, 5:36 PM | TimDickey
                              0
                            • Roverandom

                              Why does Bing bot crawl so aggressively?

                              We observer that the Bing bot is crawling our site very aggressively. We set Bing's crawl control so that it should not crawl us during heavy traffic hours, but that did not change a thing. Does anyone have the problem and even better a solution? 

                              Technical SEO | Feb 22, 2018, 10:53 AM | Roverandom
                              1
                            • DigitalAnarchy

                              Do YouTube videos in iFrames get crawled?

                              There seems to be quite a few articles out there that say iframes cause problems with organic search and that the various bots can't/won't crawl them. Most of the articles are a few years old (including Moz's video sitemap article). I'm wondering if this is still the case with YouTube/Vimeo/etc videos, all of which only offer iFrames as an embed option. I have a hard time believing that a Google property (YT) would offer an embed option that it's own bot couldn't crawl. However, let me know if that is in fact the case. Thanks! Jim

                              Technical SEO | Jan 10, 2016, 10:46 PM | DigitalAnarchy
                              0
                            • praveen439

                              Sudden drop in Rankings after 301 redirect

                              Greetings to Moz Community. Couple of months back, I have redirected my old blog to a new URL with 301 redirect because of spammy links pointed to my old blog. I have transfer all the posts manually, changed the permalink structure and 301 redirected every individual URL. All the ranking were boosted within couple of weeks and regained the traffic. After a month I have observed, the links pointed to old site are showing up in Webmaster Tools for the new domain. I was shocked (no previous experience) and again Disavowed all links. Today, all the positions went down for new domain. My questions are: 1. Did the Disavow tool worked this time with new domain? All the links pointed to old domain were devaluated? Is this the reason for ranking drop? Or 2. 301 Old domain with Unnatural links causes the issue? 3. Removing 301 will help to regain few keyword positions? I'm taking this as a case study. Already removed the 301 redirect. Looking for solid discussion.Thanks.

                              Technical SEO | Sep 22, 2014, 4:25 AM | praveen439
                              0
                            • digitalops

                              When is the last time Google crawled my site

                              How do I tell the last  time Google crawled my site. I found out it is not the "Cache" which I had thought it was.

                              Technical SEO | Feb 16, 2012, 7:06 AM | digitalops
                              0
                            • NarenBansal

                              How to stop Search Bot from crawling through a submit button

                              On our website http://www.thefutureminders.com/, we have three form fields that have three pull downs for Month, Day, and year. This is creating duplicate pages while indexing. How do we tell the search Bot to index the page but not crawl through the submit button? Thanks Naren

                              Technical SEO | Dec 12, 2011, 2:44 PM | NarenBansal
                              0
                            • sandlappercreative

                              Sudden ranking drop, no manual action

                              Sort of a strange situation I'm having and I wanted to see if I could get some thoughts. Here's what has happened... Monday morning, I realized that my website, which had been showing up at the bottom of page 2 for a specific result, had now been demoted to the bottom of page 6 (roughly a 40 spot demotion). No other keyword searches were affected. I immediately figured that this was some sort of keyword-specific penalty that I had incurred. I had done a bit of link building over the weekend (two or three directory type sites and a bio link from a site I contribute to). I also changed some anchor text on another site to match my homepage's title tag (which just so happened to be the exact phrase match I had dropped in) - I assumed this was what got me. I was slowly beginning to climb up the rankings and just got a bit impatient/overzealous. Changed the anchor text back to what it originally was and submitted a reconsideration request on Tuesday. This morning, I get the automated response in Webmaster Tools that no manual action had been taken. So my question is, would this drop have been an automated deal? If that's the case, then it's going to be mighty hard to pinpoint what I did wrong, since there's no way to know when I did whatever it was to cause the drop. Any ideas/thoughts/suggestions to regain my modest original placement?

                              Technical SEO | Nov 28, 2011, 8:58 PM | sandlappercreative
                              0

                            Get started with Moz Pro!

                            Unlock the power of advanced SEO tools and data-driven insights.

                            Start my free trial
                            Products
                            • Moz Pro
                            • Moz Local
                            • Moz API
                            • Moz Data
                            • STAT
                            • Product Updates
                            Moz Solutions
                            • SMB Solutions
                            • Agency Solutions
                            • Enterprise Solutions
                            Free SEO Tools
                            • Domain Authority Checker
                            • Link Explorer
                            • Keyword Explorer
                            • Competitive Research
                            • Brand Authority Checker
                            • MozBar Extension
                            • MozCast
                            Resources
                            • Blog
                            • SEO Learning Center
                            • Help Hub
                            • Beginner's Guide to SEO
                            • How-to Guides
                            • Moz Academy
                            • API Docs
                            About Moz
                            • About
                            • Team
                            • Careers
                            • Contact
                            Why Moz
                            • Case Studies
                            • Testimonials
                            Get Involved
                            • Become an Affiliate
                            • MozCon
                            • Webinars
                            • Practical Marketer Series
                            • MozPod
                            Connect with us

                            Contact the Help team

                            Join our newsletter
                            Moz logo
                            © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                            • Accessibility
                            • Terms of Use
                            • Privacy

                            Looks like your connection to Moz was lost, please wait while we try to reconnect.