Does it really take 3-7 days to crawl a site?
-
This is what you say on your website:
Please note: depending on the size and speed of your site, it may take between 3 and 7 days to complete your crawl.
Are you guys kidding? This is unacceptable for almost all of my deadlines, and I suspect 99% of the SEO world... you can fly to the moon faster.
-
I hear you - it's a frustration, but in a system that needs to crawl 30K+ sites each week without hitting anyone's server too hard, being a bother to a SysAdmin and keeping up with a massively complex, ever-changing queue, it's the reality.
That said, we do have a custom crawl tool you can fire anytime and usually get data back within just a few hours! It's here - http://www.seomoz.org/labs/cc
The intent/value behind the crawl inside the campaign is much less about a specific, one-time crawl, and more about having data every week, with historical information showing progress, updates and warnings (in case something goes wrong). There's lots of good free tools as well for single-purpose crawls, e.g. http://www.seomoz.org/blog/xenu-link-sleuth-more-than-just-a-broken-links-finder
Also, just to be totally clear - the system running the crawls for the web app in you campaigns is different than the Linkscape web index (which only updates every 3-4 weeks). Eventually, the two might merge, but we didn't want to bias any crawling inside Linkscape when we launched the web app last September.
-
The time it takes is not based off of your site alone, there are so may sites and links being crawled which all take processing power and bandwidth.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Crawl Test: WordPress sites with and without /feed and /trackback entires?
I have multiple WP websites and on some of the websites, on my Moz Crawl test, I see an entry for every blog post but also entries for /feed and /trackback for that single blog post. For example, www...com/someArticle www....com/someArticle/feed www...com/someArticle/trackback 1. Can anyone explain why the Crawl test is picking up the /feed and /trackback items? Is it simply because they are 301 redirects to the original post (www...com/someArticle)? 2. What setting(s) in WordPress are making this information appear? Or is it just that the site(s) that have the /feed and /trackback are displaying "normal" behavior for a WP site with a lot of trackbacks and feed entires? 3. Should /fee and /trackback, as well as /author be blocked in robots.txt? Thanks in advance for your advice and input!
Moz Pro | | Titan5520 -
Site Not Indexing & SEOMoz Reporting ZERO On-Page Report Crawls
Any help on this would be MUCH appreciated. One of my sites, aironeairsolutionsinc.com, has recently been rebuilt and the pages tweaked for some basic optimization. Based on my experience, those tweaks (geared toward keywords with relatively low competition locally) usually bump my local sites up into the top 20 or 30 at worst. 3 weeks later, it seems my site is still not indexing with Google. In addition, I AM NOTICING THAT THE ON PAGE REPORTS IN SEO MOZ ARE NOT REGISTERING THAT ANY PAGES ARE BEING CRAWLED. Again, any help from Moz staff would be awesome! :} Thanks, Ricky
Moz Pro | | RickyShockley0 -
SEO Moz analysing US site when I want it to analyse UK site
Hi there, I am trying to analyse the UK version of my website. However as the website is set to redirect people to the most relevant site and SEO Moz is based in the US, it is redirecting all the pages your crawling and therefore analysing US pages. So I am trying to look at the UK homepage but all the factors relate to the US site... Is there a way around this?? Thanks
Moz Pro | | MyTights1 -
How can I get a report of the top 500 national sites using Domain Authority and Page Authority? Does someone have to type sites in one by one on Site Explorer or is there another way?
I want to know what sites and blogs have the to rankings overall based on their Domain Authority and Page Authority - Top 500 - 1000 in nation - I want to know which ones are follow and no-follow too - Does anyone know? Has anyone run such a report yet? Thanks for help - BD
Moz Pro | | creativeguy0 -
7 Days into My Account and Only Crawled 3 Pages - What's Happened?
Am I doing something wrong? So far only 3 pages of my site have been crawled - but my account has been live for 7 days. Would it tell me if it was having trouble crawling the rest of the site?
Moz Pro | | columbus0 -
Link Diagnosis and Open Site Explorer
I work at a web design firm that's starting to offer SEO to its clients. They want to keep costs down until the SEO side becomes more established, so I've been using mostly free tools to do the SEO. I've been using a website called LinkDiagnosis.com to check backlinks for clients. It says the data is provided by SEOmoz, so I thought it must be pretty reliable. However, I just signed up for the PRO trial to test out all of SEOmoz's tools (& hopefully convince my company to sign us up), and I find that the link information between the 2 tools can be vastly different! Sometimes the number of unique linking domains is very close, but other times it can be off by hundreds, even thousands, with Open Site Explorer typically providing the higher numbers. Is the Link Diagnosis tool really powered with data from SEOmoz, or does it just receive a portion of the data, since it's a free tool? I'm just trying to figure out what will be the most reliable solution for me to keep using, and the discrepancies between the two tools has caused me to question the reliability of both. Thanks, Hector
Moz Pro | | hmunoz0 -
How to get seomoz to re-crawl a site?
I had a lot of duplicate content issues and have fixed all the other warnings. I want to check the site again.
Moz Pro | | adamzski0 -
Recent backlinks in Open Site Explorer as not showing
I saw the note today that the link index did it's monthly update, yet our site www.oznappies.com still only shows 1 linking root domain and I know there are many more links now. What do I need to do to get open Site Explorer to use the latest data? I enter our site, create the report and only see old information from 6 may 2011's link index. I have the same issue with competive link finder, links I know we have on the sites listed for our competitors are not showing for our site.
Moz Pro | | oznappies0