undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Digital Marketers
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Digital Marketers

      Simplify SEO tasks to save time and grow your traffic.

    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Technical SEO
  4. Crawl rate dropped to zero

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Crawl rate dropped to zero

Technical SEO
2
13
2.3k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • pok3rplay3r
    pok3rplay3r last edited by Mar 22, 2016, 5:05 PM

    Hello, I recently moved my site in godaddy from cpanel to managed wordpress. I bought this transfer directly from GoDaddy customer service. in this process they accidentally changed my domain from www to non www. I changed it back after the migration, but as a result of this sites craw rate from search console fell to zero and has not risen at all since then.

    In addition to this website does not display any other errors, i can ask google manually fetch my pages and it works as before, only the crawl rates seems to be dropped permanently. GoDaddy customer service also claims that do not see any errors but I think, however, that in some way they caused this during the migration when the url changed since the timing match perfectly.  also when they accidentally removed the www, crawl rate of my sites  non www version got up but fell back to zero when I changed it back to www version. Now the crawl rate of both www and non www version is zero. How do I get it to rise again? Customer service also said that the problem may be related to ftp-data of search console?  But they were not able to help any more than .Would someone from here be able to help me with this in anyway please?

    1 Reply Last reply Reply Quote 0
    • pok3rplay3r
      pok3rplay3r last edited by Apr 2, 2016, 8:41 AM Apr 2, 2016, 8:41 AM

      Hello, asnwers to the questions bolded:

      • At this rate, how long would it take Google to crawl all of your pages, (maybe it feels 10-15 is fast enough)? Over 50 days, i still cannot believe that it would be just a coincidence that crawl rate dropped so suddenly only because google suddenly thinks that my page should not be crawled that often. After all, amount of new content, quality of new links and all the other factors are much better all the time on my site, and before the drop, crawl rate increased steadily. It has to be some technical issue?
      • Has the average response time increased? If so, maybe Google feels it's overloading the server & backing off. No, it has actually went down a little bit (not much though)
      1 Reply Last reply Reply Quote 0
      • 4RS_John
        4RS_John Subscriber @pok3rplay3r last edited by Apr 1, 2016, 10:56 AM Apr 1, 2016, 10:56 AM

        Interesting. I have 2 more thoughts:

        • At this rate, how long would it take Google to crawl all of your pages, (maybe it feels 10-15 is fast enough)?
        • Has the average response time increased? If so, maybe Google feels it's overloading the server & backing off.
        1 Reply Last reply Reply Quote 0
        • pok3rplay3r
          pok3rplay3r last edited by Apr 1, 2016, 4:57 AM Apr 1, 2016, 4:57 AM

          Crawl rate still is extremely slow, average 10-15 per day except when i sent pages to be manually crawled, then it crawls those page. Before the drop the crawl rate was never under 200 per day and it was usually over 1000. anything more I can do? It seems to have no effect my rankings or anything else as l can see, but I still would like this be fixed. It has be something to do with the fact that i changed my hosting to godaddy managed wordpress hosting. but they have no clue about what could cause this. robot.txt file change seemed to have no effect or very minimum effect

          4RS_John 1 Reply Last reply Apr 1, 2016, 10:56 AM Reply Quote 0
          • topic:timeago_earlier,9 days
          • 4RS_John
            4RS_John Subscriber @pok3rplay3r last edited by Mar 23, 2016, 4:05 PM Mar 23, 2016, 4:05 PM

            Not that I'm aware of, unfortunately. Patience is an important skill when dealing with Google 😉

            1 Reply Last reply Reply Quote 0
            • pok3rplay3r
              pok3rplay3r last edited by Mar 23, 2016, 3:15 PM Mar 23, 2016, 3:15 PM

              Thanks! I will try that. I see that search console shows crawl rates with few days delay, is there somewhere i could check if it works instantly?

              4RS_John 1 Reply Last reply Mar 23, 2016, 4:05 PM Reply Quote 0
              • 4RS_John
                4RS_John Subscriber @pok3rplay3r last edited by Mar 23, 2016, 10:31 AM Mar 23, 2016, 10:31 AM

                I thought of one other possibility: Your sitemap.xml is probably auto-generated, so this shouldn't be a problem, but check to make sure that the URLs in the sitemap.xml have the www.

                Other than that I'm out of ideas - I would wait a few days to see what happens, but maybe someone else with more experience watching Google will have seen this before. If it does resolve, I'd like to know what worked.

                1 Reply Last reply Reply Quote 1
                • 4RS_John
                  4RS_John Subscriber @pok3rplay3r last edited by Mar 23, 2016, 10:29 AM Mar 23, 2016, 10:29 AM

                  I'm not convinced that robots.txt is causing your problem, but it can't hurt to change it back. In fact, while looking for instructions on how to change it I came across this blog post by Joost de Valk, (aka Yoast), that pretty much says you should remove everything that's currently in your robots.txt - and his arguments are right for everything:

                  • Blocking wp-content/plugins will stop Google from loading JS and/or CSS resources that it might need to render the page properly.
                  • Blocking wp-admin is redundant, because the wp-admin if it's linked it can still be found, and important pages already have an X-Robots HTTP header that says not to index them.

                  If you're using Yoast SEO, here are instructions on how to change the robots.txt file.

                  1 Reply Last reply Reply Quote 1
                  • pok3rplay3r
                    pok3rplay3r last edited by Mar 23, 2016, 9:17 AM Mar 23, 2016, 9:17 AM

                    Hi, one more thing. Are you 100% sure tht robot.txt file hs nothing to do with this? It changed at the sime time when the problems started to occur. It used to be :

                    User-agent: *
                    Disallow: /wp-admin/
                    Allow: /wp-admin/admin-ajax.php

                    But now it is :

                    User-agent: *
                    Crawl-delay: 1
                    Disallow: /wp-content/plugins/
                    Disallow: /wp-admin/

                    At the sime time "blocked resources" notifications started to occur in search console.

                    Blocked Resources >  Rendering without certain resources can impair the indexing of your web pages. Learn more.Status: 3/19/16152 Pages with blocked resources

                    This has to have something to do with it right?

                    4RS_John 1 Reply Last reply Mar 23, 2016, 10:29 AM Reply Quote 0
                    • pok3rplay3r
                      pok3rplay3r last edited by Mar 23, 2016, 6:56 AM Mar 23, 2016, 6:56 AM

                      Thank you for your answer, my answers bolded here below

                      • Do you see any crawl errors in the Google Search Console?  **Nothing new after the crawl rate dropped, just some old soft 404 errors and old not found errors. **
                      • If you search for your site on Google, what do you see, (does your snippet look normal)? Yes everything looks perfectly normal, just like before when the crawl rate dropped
                      • How many pages does Google say it has indexed? Is it possible it's indexed everything and is taking a break, (does it even do that?) I dont thin this is possible, since the cralw rate dropped lmost instantly from average 400 to zero after the site migration.

                      One theory is: When you moved to the non-www version of the site, Google started getting 301s redirecting it from www to non-www, and now that you've gone back to www it's getting 301s redirecting it from from non-www to www, so it's got a circular redirect. If this is the problem, how should i start to get it fixed?

                      Here's what I would do to try to kick-start indexing, if you haven't already:

                      • Make sure you have the "Preferred Domain" set to the www version of your site in_ both the www and non-www versions of your site_ in Google Search Console. Yes that is how it has been all the time
                      • In the Search Console for the www-version of your site, re-submit your sitemap. Done
                      • In the Search Console for the www-version of your site, do a Fetch as Google on your homepage, and maybe a couple of other pages, and when the Fetch is done use the option to submit those pages for indexing, (there's a monthly limit on how much of this you can do). I have done this many times since i noticed the problem, fetch as google works normally without any issues

                      Is there anything more i can do? If i want hire someone to fix this, is there any recommendations? I am not a tech guy so this is quite difficult task for me

                      4RS_John 1 Reply Last reply Mar 23, 2016, 10:31 AM Reply Quote 0
                      • 4RS_John
                        4RS_John Subscriber last edited by Mar 22, 2016, 11:22 PM Mar 22, 2016, 11:22 PM

                        I don't know why this is happening, but this is what I would check:

                        • Do you see any crawl errors in the Google Search Console?
                        • If you search for your site on Google, what do you see, (does your snippet look normal)?
                        • How many pages does Google say it has indexed? Is it possible it's indexed everything and is taking a break, (does it even do that?)

                        One theory is: When you moved to the non-www version of the site, Google started getting 301s redirecting it from www to non-www, and now that you've gone back to www it's getting 301s redirecting it from from non-www to www, so it's got a circular redirect.

                        Here's what I would do to try to kick-start indexing, if you haven't already:

                        • Make sure you have the "Preferred Domain" set to the www version of your site in both the www and non-www versions of your site in Google Search Console.
                        • In the Search Console for the www-version of your site, re-submit your sitemap.
                        • In the Search Console for the www-version of your site, do a Fetch as Google on your homepage, and maybe a couple of other pages, and when the Fetch is done use the option to submit those pages for indexing, (there's a monthly limit on how much of this you can do).

                        Good luck!

                        1 Reply Last reply Reply Quote 1
                        • 4RS_John
                          4RS_John Subscriber @pok3rplay3r last edited by Mar 22, 2016, 11:06 PM Mar 22, 2016, 11:06 PM

                          That's not so horrible - it just says not to crawl the plugins directory or the admin, and to delay a second between requests. You probably don't want your plugins or admin directories being indexed, and according to this old forum post Google ignores the crawl-delay directive, so the robots.txt isn't the problem.

                          1 Reply Last reply Reply Quote 1
                          • pok3rplay3r
                            pok3rplay3r last edited by Mar 22, 2016, 8:16 PM Mar 22, 2016, 8:16 PM

                            Hi, my robot.txt file looks like this:

                            User-agent: *
                            Crawl-delay: 1
                            Disallow: /wp-content/plugins/
                            Disallow: /wp-admin/
                            
                            This is not how it suppose to look like, right? could this cause the problem?
                            
                            4RS_John 1 Reply Last reply Mar 22, 2016, 11:06 PM Reply Quote 0
                            • 1 / 1
                            1 out of 13
                            • First post
                              1/13
                              Last post

                            Got a burning SEO question?

                            Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                            Start my free trial


                            Browse Questions

                            Explore more categories

                            • Moz Tools

                              Chat with the community about the Moz tools.

                            • SEO Tactics

                              Discuss the SEO process with fellow marketers

                            • Community

                              Discuss industry events, jobs, and news!

                            • Digital Marketing

                              Chat about tactics outside of SEO

                            • Research & Trends

                              Dive into research and trends in the search industry.

                            • Support

                              Connect on product support and feature requests.

                            • See all categories

                            Related Questions

                            • Train4Academy.co.uk

                              Customer Reviews on Product Page / Pagination / Crawl 3 review pages only

                              reviews pagination crawler disallow

                              Hi experts, I present customer feedback, reviews basically, on my website for the products that are sold. And with this comes the ability to read reviews and obviously with pagination to display the available reviews. Now I want users to be able to flick through and read the reviews to help them satisfy whatever curiosity they have. My only thinking is that the page that contains the reviews, with each click of the pagination will present roughly the same content. The only thing that changes is the title tags which will contain the number in the H1 to display the page number. I'm thinking this could be duplication but i have yet to be notified by Google in my Search console... Should i block crawlers from crawling beyond page 3 of reviews? Thanks

                              Technical SEO | Sep 23, 2023, 10:02 PM | Train4Academy.co.uk
                              0
                            • LisaBabblebird

                              Domain Authority dropped 10 points and I don't know why

                              domain authority optimization

                              In my latest site crawl, the domain authority dropped 10 points for no apparent reason. There have been no changes to the site. The only change I have made this month is to block referral spam to the site. My competitors' DAs have stayed the same too. website name: https://knowledgefront.co.uk/ Any ideas?

                              Technical SEO | Aug 9, 2022, 8:24 AM | LisaBabblebird
                              0
                            • rwat

                              Sudden Drop in Mobile Core Web Vitals

                              core web vitals google search console technical seo

                              Web Vitals Screengrab.PNG For some reason, after all URLs being previously classified as Good, our Mobile Web Vitals report suddenly shifted to the above, and it doesn't correspond with any site changes on our end. Has anyone else experience something similar or have any idea what might have caused such a shift? Curiously I'm not seeing a drop in session duration, conversion rate etc. for mobile traffic despite the seemingly sudden change.

                              Technical SEO | Dec 18, 2023, 6:45 AM | rwat
                              0
                            • iHasco

                              URLs dropping from index (Crawled, currently not indexed)

                              I've noticed that some of our URLs have recently dropped completely out of Google's index. When carrying out a URL inspection in GSC, it comes up with 'Crawled, currently not indexed'. Strangely, I've also noticed that under referring page it says 'None detected', which is definitely not the case. I wonder if it could be something to do with the following? https://www.seroundtable.com/google-ranking-index-drop-30192.html - It seems to be a bug affecting quite a few people. Here are a few examples of the URLs that have gone missing: https://www.ihasco.co.uk/courses/detail/sexual-harassment-awareness-training https://www.ihasco.co.uk/courses/detail/conflict-resolution-training https://www.ihasco.co.uk/courses/detail/prevent-duty-training Any help here would be massively appreciated!

                              Technical SEO | Oct 24, 2024, 7:05 AM | iHasco
                              0
                            • H.M.N.

                              Huge drop in rankins, traffic and impressions after changing to CloudFlare

                              Hi there, In October, one of our customer's programmer made a change on their website to optimize its loading speed. Since then, the all the SEO's metrics has dropped. Apparently, the change was to move to CloudFlare and to add Gzip compression. I was talking with the programmer and he told me he had no idea why that happened. Now comes 5 months later and the SEO metrics havn't come back yet. What seems so wierd is that two keywords in particular had the most massive drop. Those two keywords were the top keywords (more than 1k of impressions a month) and now its like there is no impressions or clics at all. Did anyone had the same event occur to them? Do you have any idea what could help this case?

                              Technical SEO | Feb 7, 2019, 8:47 PM | H.M.N.
                              0
                            • Taswirh

                              Expired domain 404 crawl error

                              I recently purchased a Expired domain from auction and after I started my new site on it, I am noticing 500+ "not found" errors in Google Webmaster Tools, which are generating from the previous owner's contents.Should I use a redirection plugin to redirect those non-exist posts to any new post(s) of my site? or I should use a 301 redirect? or I should leave them just as it is without taking further action? Please advise.

                              Technical SEO | Jul 19, 2014, 8:29 PM | Taswirh
                              1
                            • catalinmoraru

                              Blocked URL parameters can still be crawled and indexed by google?

                              Hy guys, I have two questions and one might be a dumb question but there it goes. I just want to be sure that I understand: IF I tell webmaster tools to ignore an URL Parameter, will google still index and rank my url? IS it ok if I don't append in the url structure the brand filter?, will I still rank for that brand? Thanks, PS: ok 3 questions :)...

                              Technical SEO | Jan 29, 2015, 2:23 PM | catalinmoraru
                              0
                            • evoNick

                              Crawling image folders / crawl allowance

                              We recently removed /img and /imgp from our robots.txt file thus allowing googlebot to crawl our image folders.  Not sure why we had these blocked in the first place, but we opened them up in response to an email from Google Product Search about not being able to crawl images - which can/has hurt our traffic from Google Shopping. My question is:  will allowing Google to crawl our image files eat up our 'crawl allowance'?  We wouldn't want Google to not crawl/index certain pages, and ding our organic traffic, because more of our allotted crawl bandwidth is getting chewed up crawling image files. Outside of the non-detailed crawl stat graphs from Webmaster Tools, what's the best way to check how frequently/ deeply our site is getting crawled? Thanks all!

                              Technical SEO | Apr 27, 2011, 1:21 AM | evoNick
                              0

                            Get started with Moz Pro!

                            Unlock the power of advanced SEO tools and data-driven insights.

                            Start my free trial
                            Products
                            • Moz Pro
                            • Moz Local
                            • Moz API
                            • Moz Data
                            • STAT
                            • Product Updates
                            Moz Solutions
                            • SMB Solutions
                            • Agency Solutions
                            • Enterprise Solutions
                            Free SEO Tools
                            • Domain Authority Checker
                            • Link Explorer
                            • Keyword Explorer
                            • Competitive Research
                            • Brand Authority Checker
                            • Local Citation Checker
                            • MozBar Extension
                            • MozCast
                            Resources
                            • Blog
                            • SEO Learning Center
                            • Help Hub
                            • Beginner's Guide to SEO
                            • How-to Guides
                            • Moz Academy
                            • API Docs
                            About Moz
                            • About
                            • Team
                            • Careers
                            • Contact
                            Why Moz
                            • Case Studies
                            • Testimonials
                            Get Involved
                            • Become an Affiliate
                            • MozCon
                            • Webinars
                            • Practical Marketer Series
                            • MozPod
                            Connect with us

                            Contact the Help team

                            Join our newsletter

                            Access all your tools in one place. Whether you're tracking progress or analyzing data, everything you need is at your fingertips.

                            Moz logo
                            © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                            • Accessibility
                            • Terms of Use
                            • Privacy

                            Looks like your connection to Moz was lost, please wait while we try to reconnect.