MOZ Crawler
-
Hi,
how much time it will take MOZ crawler to take entire site? In 24 hours it crawled only 500 pages isn't it too slow? My website has almost 50k pages.
-
Hi there,
Sam from Moz's Help Team here! Could you pop a message over to help@moz.com about this so we can ensure everything is running smoothly for you?
Thank you!
-
Yeah it will take a while but it's necessarily slow. Moz's crawler has to share bandwidth with your real, human users. If they crawl your site too fast they could lag your site or accidentally DDoS you or something
Since it's a paid for service, I'm guessing that Moz err on the side of caution with Rogerbot and don't crawl insanely fast. I know that Rogerbot supports adding a crawl delay to make the crawling even slower:
https://moz.com/help/moz-procedures/crawlers/rogerbot
But I don't think there are any 'go faster' commands. Whilst I really like Moz's crawler I tend to find that the SEMRush one is faster for on-site stuff, though it does have URL crawling caps
To be honest, just get Screaming Frog and learn how to use that. Now that it has a database storage mode (instead of maniacally trying to store all the crawl data in the RAM) - it probably could crawl all 50K URLs. It could probably do it faster than Rogerbot, though you wouldn't wanna go nutty as you could easily DDoS yourself :')
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved The Moz.com bot is overloading my server
0 -
How do I combat title tag issues in Moz because of long company/site name? [Currently using Yoast SEO Plugin]
I've only just started to dip my toe into the SEO pool, so please forgive me if this is a silly question. I've been working heavily in the Moz tool and the Yoast SEO plugin to clean up the meta descriptions on our site, as there were many pages without descriptions in place (we just transferred from a different CMS to Wordpress within the last month or so). Now I'm noticing on our Site Crawl that we have 60+ pages that have title elements that are considered too long. When I go in to fix these issues on our blog posts (as this is where most of the issues lie), Yoast is already giving me the green light on the title lengths. I've discovered that the reason why Moz is telling us that many of our site lengths are too long is because "| Parker Staffing Services" is added to the end of our title tags automatically. Deep breath Now that you've heard all of the backstory... Is there a way that I can prevent that site title (which is 26 characters all by itself! Talk about limiting!) from displaying on search engines when our blog posts or pages show up in search results? I've poked around in the Titles & Metas settings that come with Yoast, but I fear making too many changes will result in me screwing something up. Help an SEO beginner out? 🙂
Moz Pro | | giaciccone0 -
Abnormal crawl issues appearing in my Moz results
I have been asked to look at a site for a friend and was more than surprised to see 16,9k crawl issues appear in the dashboard... of this 6,238 are duplicate page content and 5878 are duplicated page titles. What on earth is going on? I have spoken to the web developer as it appears there is a dev site somewhere and this is his response [Can I stress that Google determines which site was in the index first and then removes other sites it sees as having duplicate content. Our dev sites appearing in the search index would not affect your ranking due to duplicate content as Google would see your site as the first site with the content] As I cannot make contact with him, I am scratching my head, surely a dev site should be no-indexed, it sounds as though he is saying that its ok because Google will take the main site as the first site with the content... Very confused! Help need MOZ community. Manythanks, Sarah
Moz Pro | | Mutatio_Digital0 -
Problem to log into moz
Every time the moz logs me out from the account and then I can not log in. It shows on the left side my name like I am logged in and then when I want go to community suddenly I am not logged in. It offen shows 502 error. It was first doing on firefox, then I manage to log in chrome and now I had to log in private browsing.
Moz Pro | | Rebeca11 -
Is the Moz Ranking Report correct?
Hello here, today I got my weekly report from Moz by email and it reported a strong decline of rankings on some of our major keywords. Then I went to check on Google personally, and instead I found my pages ranked much higher than reported by Moz (please note that my results wasn't personalized...). I also use a desktop program to check my rankings, and what I found personally on Google corresponded to what my desktop program reported me. Here is an example: my website is virtualsheetmusic.com, and Moz reported a rank NOT in top 50 for the following keyword: "moonlight sonata sheet music" Whereas we are actually on the 19th spot. Another example for the keyword "czardas violin sheet music", Moz reports NOT in top 50, whereas we are actually at the 5th spot on the first page of Google results! Why's this? Anyone can explain? Is there anything wrong with Moz ranking report?
Moz Pro | | fablau0 -
Annotations in Moz tools?
Beginner question, I know, but is there a way to add an annotation or event to the Moz tools, as I can in Google Analytics? I'd like to mark when a change was made. Thanks.
Moz Pro | | KelloggMoz0 -
SEOmoz crawler not crawling my site
We set up a new campaign in SEOmoz on Friday. It is my understanding that the preliminary crawl can cover up to 250 and this has been our experience in the past. However, the preliminary crawl only went through 2 pages. This is a larger eCommerce site with many pages. Any ideas why more pages weren't crawled? We set up the campaign to track at the root domain level.
Moz Pro | | IMM0