MOZ Crawler
-
Hi,
how much time it will take MOZ crawler to take entire site? In 24 hours it crawled only 500 pages isn't it too slow? My website has almost 50k pages.
-
Hi there,
Sam from Moz's Help Team here! Could you pop a message over to help@moz.com about this so we can ensure everything is running smoothly for you?
Thank you!
-
Yeah it will take a while but it's necessarily slow. Moz's crawler has to share bandwidth with your real, human users. If they crawl your site too fast they could lag your site or accidentally DDoS you or something
Since it's a paid for service, I'm guessing that Moz err on the side of caution with Rogerbot and don't crawl insanely fast. I know that Rogerbot supports adding a crawl delay to make the crawling even slower:
https://moz.com/help/moz-procedures/crawlers/rogerbot
But I don't think there are any 'go faster' commands. Whilst I really like Moz's crawler I tend to find that the SEMRush one is faster for on-site stuff, though it does have URL crawling caps
To be honest, just get Screaming Frog and learn how to use that. Now that it has a database storage mode (instead of maniacally trying to store all the crawl data in the RAM) - it probably could crawl all 50K URLs. It could probably do it faster than Rogerbot, though you wouldn't wanna go nutty as you could easily DDoS yourself :')
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I am using moz pro but ı can't use keyword explorer
hi am buying moz pro but can't please help me. ı want scan this web site https://baliktasarim.com/
Moz Pro | | nurmasumi20020 -
Moz Insights are raising alarms --Are they useful?
Hi, Moz is alerting us about URLs being too long, titles being too long, and duplicate content. These are mostly for pages with categories of content, or the author bio pages --pages that are included by default with the template we're using in HubSpot. So that there isn't a quick way to make these changes, and anyway the pages are low-value.So my question is, how much are these issues impacting our SEO and domain authority?If these are pages no one visits (like the category and author pages) is there any reason this would be harming our SEO efforts?Thanks!
Moz Pro | | ASCI-Marketing0 -
Domain authority get down significantly. Internal MOZ Issue? Google Algoritm Update?
I've noticed that several websites have lost DA since middle of october. I attached a picture. Could be this a MOZ internal issue like happened a time ago? Not only the DA get down, I have lost C-blocks links, too
Moz Pro | | NachoRetta
I check in http://health.moz.com/ and found this Crawl Services Outage: http://health.moz.com/incidents/fbn00m7xxzx2
But not appear to have direct relation with my question.
Or may be a google algoritm update? 3anVE5J.png2 -
How do I know which keywords to track in my MOZ Campaigns?
Good morning Mozzers from the London, England. This may be a silly question to many of the community, but here goes "How do I know which keywords to track in my MOZ campaigns?" I work in the gyms section in the UK and am tracking keywords like "Gyms in Liverpool" or "Manchester gyms" which send me some traffic. I can see search terms like the above do send me traffic from Google's Webmaster Tools, but obviously this tool won't highlight the keyword opportunities I am missing out on because I currently don't rank. Any guidance on this dilemma would be hugely appreciated, Kindest regards Ben
Moz Pro | | Bendall0 -
I need an interlinking report for my site, is there a report in Moz or another application that tell me how all of my pages are linked to other pages on my site?
I am in the process of doing a redesign for one of my sites. I need an interlinking report for my site. Is there a report in Moz or another application that tell me how all of my pages are linked to other pages on my site?
Moz Pro | | seoflorida0 -
@staff When is Moz Analytics coming out?
I want to see a video demo or something for the new Moz Analytics. When is the software going to be available for us subscribers?
Moz Pro | | Francisco_Meza0 -
GWT data v SEO Moz data
I'm sure there's a simple explanation but I can't find it- Looking through the GWT avg position data v SEOMoz ranking data and Market Samurai data there is a large discrepancy. For one term that we hold #1, and have held for a month or two GWTis showing that the avg position is 180 (and down 30). Any reason you know of why the GWT avg position data would be so out of whack with our SERP's and other data? Thanks.
Moz Pro | | MrPaulB0 -
Moz Toolbar Analyzer - Links
On the moz toolbar there is an Analyze tool. The third tab is for viewing Page Attributes. That page contains a section called On Page Links. I can see the External Link count, but I need to drill down further to look at these external links. I have viewed a page and don't see the external links this tool is seeing. I have searched the page, the source code, and used other tools on the page. The numbers don't match up. Is there a tool which lists all the external links on a given page?
Moz Pro | | RyanKent0