Does it really take 3-7 days to crawl a site?
-
This is what you say on your website:
Please note: depending on the size and speed of your site, it may take between 3 and 7 days to complete your crawl.
Are you guys kidding? This is unacceptable for almost all of my deadlines, and I suspect 99% of the SEO world... you can fly to the moon faster.
-
I hear you - it's a frustration, but in a system that needs to crawl 30K+ sites each week without hitting anyone's server too hard, being a bother to a SysAdmin and keeping up with a massively complex, ever-changing queue, it's the reality.
That said, we do have a custom crawl tool you can fire anytime and usually get data back within just a few hours! It's here - http://www.seomoz.org/labs/cc
The intent/value behind the crawl inside the campaign is much less about a specific, one-time crawl, and more about having data every week, with historical information showing progress, updates and warnings (in case something goes wrong). There's lots of good free tools as well for single-purpose crawls, e.g. http://www.seomoz.org/blog/xenu-link-sleuth-more-than-just-a-broken-links-finder
Also, just to be totally clear - the system running the crawls for the web app in you campaigns is different than the Linkscape web index (which only updates every 3-4 weeks). Eventually, the two might merge, but we didn't want to bias any crawling inside Linkscape when we launched the web app last September.
-
The time it takes is not based off of your site alone, there are so may sites and links being crawled which all take processing power and bandwidth.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why my site not crawl?
hi all me allow rogerbot in robots.txt but rogerbot can't crawl my site my site: toska-co.ir
Moz Pro | | jahanidawodi0 -
7,608 High Priority Crawl Diagnostic problems
Hey There, I have an e-commerce site that is showing 7,608 High Priorities to fix - 7,536 are duplicate content. What's the most effective process to start with? I'm open to outsourcing some of the work to an expert - email me on dave@emanbee.com Thanks for your time, Dave
Moz Pro | | emanbee0 -
How to track data from old site and new site with the same URL?
We are launching a new site within the next 48 hours. We have already purchased the 30 day trial and we will continue to use this tool once the new site is launched. Just looking for some tips and/or best practices so we can compare the old data vs. the new data moving forward....thank you in advance for your response(s). PB3
Moz Pro | | Issuer_Direct0 -
403 error for a member site
Perhaps a stupid question but SEOmoz registers 403 errors for pages behind a membersite (ie. they are restricted on purpose). Should I noindex these pages or just let SEOmoz register these "errors"?
Moz Pro | | Crunchii0 -
What do I really need
Hello all. I’m new to this SEO stuff and after reading much, remain very, very confused as to what I really need. First, I’ll say I’m here because I have been impressed with the posts and the general professional approach that seem to exist on Seomoz. My problem is that I run a small business that’s not doing so well and I do know that my website needs to be found. With this being the case I contacted an SEO company and on being given a price for 12 months of service, almost feel of my chair as this would clear me out. This, without any guarantee. Due to the above I am now considering having a go at the SEO myself. However, this leads to another problem in that, given it appears I need so many online services, it again becomes very expensive for one man and his single website. I have seen much written about Seomoz, Raven (something I’m trying now), Seospyglass and Seo Powersuite (a few reports appear bad) and between these and other possibly lesser well known brands leaves me wondering what I need to achieve improved website visibility. Does not one product exist that can help the smaller business without breaking the bank, and if not, what might be the best approach to tackle this problems cost effectively. I am aware that Seomoz has it’s followers, but having said this, it appears that a good degree of openness and honesty is provided here. Your comments would be most appreciated.
Moz Pro | | AshJez0 -
Backlinks using Open Site Explorer
Hey guys, So I was doing some backlink analysis for a client and was utilizing Open Site Explorer, which stated that I had roughly 2,111 External Followed Links. However, I can't seem to find a way to export all 2,111 backlinks. I wanted to make sure if the links I had acquired had been indexed and crawled, and picked up, but I can't seem to access the full list on a domain level, only on the subdomain level. Anybody have any suggestions or advice? Any would be helpful! Best, James
Moz Pro | | sixspokemedia0 -
Is there such thing as a site free from errors?
Or is this a given? I am new to SEO and SEOmoz. One of my campaigns is completley free of errors...the others are a work in progress. Now I realize that SEO is never done, but can a site actually be free of errors? If so... I just gave myself a pat on the back.
Moz Pro | | AtoZion0