Does it really take 3-7 days to crawl a site?
-
This is what you say on your website:
Please note: depending on the size and speed of your site, it may take between 3 and 7 days to complete your crawl.
Are you guys kidding? This is unacceptable for almost all of my deadlines, and I suspect 99% of the SEO world... you can fly to the moon faster.
-
I hear you - it's a frustration, but in a system that needs to crawl 30K+ sites each week without hitting anyone's server too hard, being a bother to a SysAdmin and keeping up with a massively complex, ever-changing queue, it's the reality.
That said, we do have a custom crawl tool you can fire anytime and usually get data back within just a few hours! It's here - http://www.seomoz.org/labs/cc
The intent/value behind the crawl inside the campaign is much less about a specific, one-time crawl, and more about having data every week, with historical information showing progress, updates and warnings (in case something goes wrong). There's lots of good free tools as well for single-purpose crawls, e.g. http://www.seomoz.org/blog/xenu-link-sleuth-more-than-just-a-broken-links-finder
Also, just to be totally clear - the system running the crawls for the web app in you campaigns is different than the Linkscape web index (which only updates every 3-4 weeks). Eventually, the two might merge, but we didn't want to bias any crawling inside Linkscape when we launched the web app last September.
-
The time it takes is not based off of your site alone, there are so may sites and links being crawled which all take processing power and bandwidth.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Crawl Test error
Moz crawl test show blank report for my website test - guitarcontrol.com. Why??? Please suggest.
Moz Pro | | zoe.wilson170 -
Duplicate title on Crawl
Ok, this should hopefully be a simple one. Not sure if this is a Moz crawl issue of redirect issue. Moz is reporting duplicate title for www.site.co.uk , site.co.uk and www.site.co.uk/home.aspx is this a canonical change or a moz setting I need to get this number lower.
Moz Pro | | smartcow0 -
Get into Google : New Sites
I have a brand new website. It was created 10 days ago. How long would it take for it to show up in search results? I understand that since the site is new, there are no sites sending it backlinks. Also, i have optimized the page for my keyword "xyz" and it received an A grade. The site does not figure even in the top 50 results. Please help me out. It is a one page web application that needs to drive traffic to survive.
Moz Pro | | dl_s0 -
Rogerbot did not crawl my site ! What might be the problem?
When I saw the new crawl for my site I wondered why there are no errors, no warning and 0 notices anymore. Then I saw that only 1 page was crawled. There are no Error Messages or webmasters Tools also did not report anything about crawling problems. What might be the problem? thanks for any tips!
Moz Pro | | inlinear
Holger rogerbot-did-not-crawl.PNG0 -
Slowing down SEOmoz Crawl Rate
Is there a way to slow down SEOmoz crawl rate? My site is pretty huge and I'm getting 10k pages crawled every week, which is great. However I sometimes get multiple page requests in one second which slows down my site a bit. If this feature exists I couldn't find it, if it doesn't, it's a great idea to have, in a similar way to how Googlebot do it. Thanks.
Moz Pro | | corwin0 -
Data from Open Site Explorer to Excel
Hello, Im having a problem with the data I pull off from Open Site Explorer. Everytime I download a report in CSV, when I open it in Excel 2007, all the information is like this: http://i.imgur.com/rwMxO.png What can I do to extract the information exactly as it appears in the Open Site Explorer with all the fields in the right place? Tks guys, Regards, Pedro Pereira [
Moz Pro | | PedroM](<a href=)" target="_blank">a> rwMxO.png
0