Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • Case Studies
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      What is your Brand Authority?
      Moz

      What is your Brand Authority?

      Check yours now
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • SEO Q&A

        Insights & discussions from an SEO community of 500,000+.

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • Case Studies

        Explore how Moz drives ROI with a proven track record of success.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Technical SEO
    4. Crawl rate dropped to zero

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Crawl rate dropped to zero

    Technical SEO
    2
    13
    2272
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • pok3rplay3r
      pok3rplay3r last edited by

      Hello, I recently moved my site in godaddy from cpanel to managed wordpress. I bought this transfer directly from GoDaddy customer service. in this process they accidentally changed my domain from www to non www. I changed it back after the migration, but as a result of this sites craw rate from search console fell to zero and has not risen at all since then.

      In addition to this website does not display any other errors, i can ask google manually fetch my pages and it works as before, only the crawl rates seems to be dropped permanently. GoDaddy customer service also claims that do not see any errors but I think, however, that in some way they caused this during the migration when the url changed since the timing match perfectly.  also when they accidentally removed the www, crawl rate of my sites  non www version got up but fell back to zero when I changed it back to www version. Now the crawl rate of both www and non www version is zero. How do I get it to rise again? Customer service also said that the problem may be related to ftp-data of search console?  But they were not able to help any more than .Would someone from here be able to help me with this in anyway please?

      1 Reply Last reply Reply Quote 0
      • pok3rplay3r
        pok3rplay3r last edited by

        Hello, asnwers to the questions bolded:

        • At this rate, how long would it take Google to crawl all of your pages, (maybe it feels 10-15 is fast enough)? Over 50 days, i still cannot believe that it would be just a coincidence that crawl rate dropped so suddenly only because google suddenly thinks that my page should not be crawled that often. After all, amount of new content, quality of new links and all the other factors are much better all the time on my site, and before the drop, crawl rate increased steadily. It has to be some technical issue?
        • Has the average response time increased? If so, maybe Google feels it's overloading the server & backing off. No, it has actually went down a little bit (not much though)
        1 Reply Last reply Reply Quote 0
        • 4RS_John
          4RS_John Subscriber @pok3rplay3r last edited by

          Interesting. I have 2 more thoughts:

          • At this rate, how long would it take Google to crawl all of your pages, (maybe it feels 10-15 is fast enough)?
          • Has the average response time increased? If so, maybe Google feels it's overloading the server & backing off.
          1 Reply Last reply Reply Quote 0
          • pok3rplay3r
            pok3rplay3r last edited by

            Crawl rate still is extremely slow, average 10-15 per day except when i sent pages to be manually crawled, then it crawls those page. Before the drop the crawl rate was never under 200 per day and it was usually over 1000. anything more I can do? It seems to have no effect my rankings or anything else as l can see, but I still would like this be fixed. It has be something to do with the fact that i changed my hosting to godaddy managed wordpress hosting. but they have no clue about what could cause this. robot.txt file change seemed to have no effect or very minimum effect

            4RS_John 1 Reply Last reply Reply Quote 0
            • 4RS_John
              4RS_John Subscriber @pok3rplay3r last edited by

              Not that I'm aware of, unfortunately. Patience is an important skill when dealing with Google 😉

              1 Reply Last reply Reply Quote 0
              • pok3rplay3r
                pok3rplay3r last edited by

                Thanks! I will try that. I see that search console shows crawl rates with few days delay, is there somewhere i could check if it works instantly?

                4RS_John 1 Reply Last reply Reply Quote 0
                • 4RS_John
                  4RS_John Subscriber @pok3rplay3r last edited by

                  I thought of one other possibility: Your sitemap.xml is probably auto-generated, so this shouldn't be a problem, but check to make sure that the URLs in the sitemap.xml have the www.

                  Other than that I'm out of ideas - I would wait a few days to see what happens, but maybe someone else with more experience watching Google will have seen this before. If it does resolve, I'd like to know what worked.

                  1 Reply Last reply Reply Quote 1
                  • 4RS_John
                    4RS_John Subscriber @pok3rplay3r last edited by

                    I'm not convinced that robots.txt is causing your problem, but it can't hurt to change it back. In fact, while looking for instructions on how to change it I came across this blog post by Joost de Valk, (aka Yoast), that pretty much says you should remove everything that's currently in your robots.txt - and his arguments are right for everything:

                    • Blocking wp-content/plugins will stop Google from loading JS and/or CSS resources that it might need to render the page properly.
                    • Blocking wp-admin is redundant, because the wp-admin if it's linked it can still be found, and important pages already have an X-Robots HTTP header that says not to index them.

                    If you're using Yoast SEO, here are instructions on how to change the robots.txt file.

                    1 Reply Last reply Reply Quote 1
                    • pok3rplay3r
                      pok3rplay3r last edited by

                      Hi, one more thing. Are you 100% sure tht robot.txt file hs nothing to do with this? It changed at the sime time when the problems started to occur. It used to be :

                      User-agent: *
                      Disallow: /wp-admin/
                      Allow: /wp-admin/admin-ajax.php

                      But now it is :

                      User-agent: *
                      Crawl-delay: 1
                      Disallow: /wp-content/plugins/
                      Disallow: /wp-admin/

                      At the sime time "blocked resources" notifications started to occur in search console.

                      Blocked Resources >  Rendering without certain resources can impair the indexing of your web pages. Learn more.Status: 3/19/16152 Pages with blocked resources

                      This has to have something to do with it right?

                      4RS_John 1 Reply Last reply Reply Quote 0
                      • pok3rplay3r
                        pok3rplay3r last edited by

                        Thank you for your answer, my answers bolded here below

                        • Do you see any crawl errors in the Google Search Console?  **Nothing new after the crawl rate dropped, just some old soft 404 errors and old not found errors. **
                        • If you search for your site on Google, what do you see, (does your snippet look normal)? Yes everything looks perfectly normal, just like before when the crawl rate dropped
                        • How many pages does Google say it has indexed? Is it possible it's indexed everything and is taking a break, (does it even do that?) I dont thin this is possible, since the cralw rate dropped lmost instantly from average 400 to zero after the site migration.

                        One theory is: When you moved to the non-www version of the site, Google started getting 301s redirecting it from www to non-www, and now that you've gone back to www it's getting 301s redirecting it from from non-www to www, so it's got a circular redirect. If this is the problem, how should i start to get it fixed?

                        Here's what I would do to try to kick-start indexing, if you haven't already:

                        • Make sure you have the "Preferred Domain" set to the www version of your site in_ both the www and non-www versions of your site_ in Google Search Console. Yes that is how it has been all the time
                        • In the Search Console for the www-version of your site, re-submit your sitemap. Done
                        • In the Search Console for the www-version of your site, do a Fetch as Google on your homepage, and maybe a couple of other pages, and when the Fetch is done use the option to submit those pages for indexing, (there's a monthly limit on how much of this you can do). I have done this many times since i noticed the problem, fetch as google works normally without any issues

                        Is there anything more i can do? If i want hire someone to fix this, is there any recommendations? I am not a tech guy so this is quite difficult task for me

                        4RS_John 1 Reply Last reply Reply Quote 0
                        • 4RS_John
                          4RS_John Subscriber last edited by

                          I don't know why this is happening, but this is what I would check:

                          • Do you see any crawl errors in the Google Search Console?
                          • If you search for your site on Google, what do you see, (does your snippet look normal)?
                          • How many pages does Google say it has indexed? Is it possible it's indexed everything and is taking a break, (does it even do that?)

                          One theory is: When you moved to the non-www version of the site, Google started getting 301s redirecting it from www to non-www, and now that you've gone back to www it's getting 301s redirecting it from from non-www to www, so it's got a circular redirect.

                          Here's what I would do to try to kick-start indexing, if you haven't already:

                          • Make sure you have the "Preferred Domain" set to the www version of your site in both the www and non-www versions of your site in Google Search Console.
                          • In the Search Console for the www-version of your site, re-submit your sitemap.
                          • In the Search Console for the www-version of your site, do a Fetch as Google on your homepage, and maybe a couple of other pages, and when the Fetch is done use the option to submit those pages for indexing, (there's a monthly limit on how much of this you can do).

                          Good luck!

                          1 Reply Last reply Reply Quote 1
                          • 4RS_John
                            4RS_John Subscriber @pok3rplay3r last edited by

                            That's not so horrible - it just says not to crawl the plugins directory or the admin, and to delay a second between requests. You probably don't want your plugins or admin directories being indexed, and according to this old forum post Google ignores the crawl-delay directive, so the robots.txt isn't the problem.

                            1 Reply Last reply Reply Quote 1
                            • pok3rplay3r
                              pok3rplay3r last edited by

                              Hi, my robot.txt file looks like this:

                              User-agent: *
                              Crawl-delay: 1
                              Disallow: /wp-content/plugins/
                              Disallow: /wp-admin/
                              
                              This is not how it suppose to look like, right? could this cause the problem?
                              
                              4RS_John 1 Reply Last reply Reply Quote 0
                              • 1 / 1
                              • First post
                                Last post

                              Got a burning SEO question?

                              Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                              Start my free trial


                              Browse Questions

                              Explore more categories

                              • Moz Tools

                                Chat with the community about the Moz tools.

                              • SEO Tactics

                                Discuss the SEO process with fellow marketers

                              • Community

                                Discuss industry events, jobs, and news!

                              • Digital Marketing

                                Chat about tactics outside of SEO

                              • Research & Trends

                                Dive into research and trends in the search industry.

                              • Support

                                Connect on product support and feature requests.

                              • See all categories

                              Related Questions

                              • Tz_Seo

                                How google crawls images and which url shows as source?

                                Hi, I noticed that some websites host their images to a different url than the one their actually website is hosted but in the end google link to the one that the site is hosted. Here is an example: This is a page of a hotel in booking.com: http://www.booking.com/hotel/us/harrah-s-caesars-palace.en-gb.html When I try a search for this hotel in google images it shows up one of the images of the slideshow. When I click on the image on Google search, if I choose the Visit Page button it links to the url above but the actual image is located in a totally different url: http://r-ec.bstatic.com/images/hotel/840x460/135/13526198.jpg My question is can you host your images to one site but show it to another site and in the end google will lead to the second one?

                                Technical SEO | | Tz_Seo
                                0
                              • danatanseo

                                Bingbot appears to be crawling a large site extremely frequently?

                                Hi All! What constitutes a normal crawl rate for daily bingbot server requests for large sites? Are any of you noticing spikes in Bingbot crawl activity? I did find a "mildly" useful thread at Black Hat World containing this quote: "The reason BingBot seems to be terrorizing your site is because of your site's architecture; it has to be misaligned. If you are like most people, you paid no attention to setting up your website to avoid this glitch. In the article referenced by Oxonbeef, the author's issue was that he was engaging in dynamic linking, which pretty much put the BingBot in a constant loop. You may have the same type or similar issue particularly if you set up a WP blog without setting the parameters for noindex from the get go." However, my gut instinct says this isn't it and that it's more likely that someone or something is spoofing bingbot. I'd love to hear what you guys think! Dana

                                Technical SEO | | danatanseo
                                1
                              • Editor-FabiusMaximus_Website

                                Google stopped crawling my site. Everybody is stumped.

                                This has stumped the Wordpress staff and people in the Google Webmasters forum. We are in Google News (have been for years), and so new posts are crawled immediately. On Feb 17-18 Crawl Stats dropped 85%, and new posts were no longer indexed (not appearing on News or search). Data highlighter attempts return "This URL could not be found in Google's index." No manual actions by Google. No changes to the website; no custom CSS. No Site Errors or new URL errors. No sitemap problems (resubmitting didn't help). We're on wordpress.com, so no odd code. We can see the robot.txt file. Other search engines can see us, as can social media websites. Older posts still index, but loss of News is a big hit. Also, I think overall Google referrals are dropping. We can Fetch the URL for a new post, and many hours later it appears on Google and News, and we can then use Data Highlighter. It's now 6 days and no recovery. Everybody is stumped. Any ideas? I just joined, so this might be the wrong venue. If so, apologies.

                                Technical SEO | | Editor-FabiusMaximus_Website
                                0
                              • zenstorageunits

                                Sitemap indexed pages dropping

                                About a month ago I noticed my pages indexed from my sitemap are dropping.There are 134 pages in my sitemap and only 11 are indexed. It used to be 117 pages and just died off quickly. I still seem to be getting consistant search traffic but I'm just not sure whats causing this. There are no warnings or manual actions required in GWT that I can find.

                                Technical SEO | | zenstorageunits
                                0
                              • MeMediaSEO

                                Google Impressions Drop Due to Expired SSL

                                Recently I noticed a huge drop in our clients Google Impressions via GWMT from 900 impressions to 70 overnight on October 30, 2012 and has remained this way for the entire month of November 2012. The SSL Cert had expired in mid October due to the notification message for renewal going to the SPAM folder and being missed. Is it possible for an SSL expiry to be related to this massive drop in daily impressions which in-turn has also effected traffic? I also can't see any evidence of duplicate pages (ie. https and http) being indexed but to be honest I'm not the one doing the SEO therefore haven't been tracking this. Thanks for your help! Chris

                                Technical SEO | | MeMediaSEO
                                0
                              • guycampbell

                                Drop Down Menu - Link Juice Depletion

                                Hi, We have a site with 7 top level sections all of which contain a large number of subsections which may then contain further sub sections. To try and ensure the best user experience we have a top navigation with the 7 top level sections and when hovered a selection of the key sub sections. Although I like this format for the user as it makes it easier for them to find the most important sections / sub sections it does lead to a lot of links within every page on the site. In general each top section has a drop down with approx 10 - 15 subsections. This has therefore lead to SeoMoz's tools issuing its too many internal links warning. Then alongside this I am left wondering if I shouldn’t have to many links to my subsections and whether I would be better off being more selective of when I link to them. For instance I could choose the top 5 sub sections and place a link to them from our homepage and by doing so I would be passing a greater amount of link juice down the line. So I guess my dilemma is between ensuring the user has as easy a time traversing the site as possible whilst I try to keep a close watch on where, and how, our link juice is distributed. One solution I am considering is whether no-follow links could be utilised within the drop down menus? This way I could then have the desired user navigation and I would be in greater control of what pages link to which sub sections. Would that even work? Any advice would be greatly appreciated, Regards, Guy

                                Technical SEO | | guycampbell
                                1
                              • digitalops

                                When is the last time Google crawled my site

                                How do I tell the last  time Google crawled my site. I found out it is not the "Cache" which I had thought it was.

                                Technical SEO | | digitalops
                                0
                              • rosstaylor

                                Bing rank drop off for multiple sites

                                Hi Mozzers, Seeing some wacky stuff going on on some sites I manage. In more than a few, the ranking on bing has dropped basically overnight from page one spots to not being found on the first 100 positions. Anyone else seeing similar results? Some of the sites are fairly new, some have been around for ages, some are wordpress, some are not. I've been searching for some news of a big change on bing, but keep reading about bing dropping the thin sites during black friday. In one example, I had the site set up in BWT for a while, and had a look at the data. The reports show that the pages are crawled, the index summary shows pages indexed, and there seems to be no crawl errors, but rankings are absolutely gone. Also, I can't see the sites in bing if I search "site:example.com" in bing. Here's 2 examples, the first would make sense since it's pretty thin as I havent added much content yet: http://homewindowtint.org but this one doesn't make sense to me. Sure there's a few errors, but to be dropped like a rock seems weird http://www.ahmedandsukaram.com

                                Technical SEO | | rosstaylor
                                0

                              Get started with Moz Pro!

                              Unlock the power of advanced SEO tools and data-driven insights.

                              Start my free trial
                              Products
                              • Moz Pro
                              • Moz Local
                              • Moz API
                              • Moz Data
                              • STAT
                              • Product Updates
                              Moz Solutions
                              • SMB Solutions
                              • Agency Solutions
                              • Enterprise Solutions
                              Free SEO Tools
                              • Domain Authority Checker
                              • Link Explorer
                              • Keyword Explorer
                              • Competitive Research
                              • Brand Authority Checker
                              • MozBar Extension
                              • MozCast
                              Resources
                              • Blog
                              • SEO Learning Center
                              • Help Hub
                              • Beginner's Guide to SEO
                              • How-to Guides
                              • Moz Academy
                              • API Docs
                              About Moz
                              • About
                              • Team
                              • Careers
                              • Contact
                              Why Moz
                              • Case Studies
                              • Testimonials
                              Get Involved
                              • Become an Affiliate
                              • MozCon
                              • Webinars
                              • Practical Marketer Series
                              • MozPod
                              Connect with us

                              Contact the Help team

                              Join our newsletter
                              Moz logo
                              © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                              • Accessibility
                              • Terms of Use
                              • Privacy

                              Looks like your connection to Moz was lost, please wait while we try to reconnect.