Adjusting SEOmoz Crawling Speed
-
How do you adjust the SEOmoz crawling speed?
SEOmoz tried to crawl 10,000 pages in 3 hours and crashed our MySQL server.
-
Hello,
You can add a crawl delay to your robots.txt file.
It would look something like this:
User-Agent: rogerbot
Crawl-Delay: 5(time is in seconds between page requests)
Hope that helps!
Kenny
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Crawl Test error
Moz crawl test show blank report for my website test - guitarcontrol.com. Why??? Please suggest.
Moz Pro | | zoe.wilson170 -
No follow links also been reported in SEOmoz crawl diagnostics
Hi, Why does SEOmoz reports links which has been marked as 'nofollow'. I am getting 'Overly-Dynamic URL' reports on links which I have designated as nofollow which means Google will discount them. So why does SEOmoz still report them. Thanks.
Moz Pro | | malpani0 -
SEOmoz Not Picking Up Links
I'm finding that SEOmoz often doesn't pick up on many of the links my client's websites are attaining. The links do appear in search engine and often have a high domain authority (often PR links, which I why I follow up to see if they've been picked up). I do tend to see many of the links appear in webmaster tools, so it's a little frustrating that they don't appear, even after months of pages being live. Thoughts?
Moz Pro | | SebastianDyer0 -
Still Cant Crawl My Site
I've removed all blocks but two from our htaccess. They are for amazonaws.com to block amazon from crawling us. I did a fetch as google in our WM tools on our robots txt with success. SEOMoz crawler here hit's our site and gets a 403. I've looks in our blocked request logs and amazon is the only one in there. What is going on here?
Moz Pro | | martJ0 -
The crawl report shows a lot of 404 errors
They are inactive products, and I can't find any active links to these product pages. How can I tell where the crawler found the links?
Moz Pro | | shopwcs0 -
SEOMoz Link Analysis Not Updating?
Hi there, I am wondering why the Link Analysis in SEOMoz takes so long to recognise new back-links. I have had the same figures showing for months when I know that I have added a lot more backlinks that are showing in my Google WebMaster account and I have also proven exist by searching directly in Google's search engine. How often does the Link Analysis section update itself? Are these figures worth following at all or are they useless? Thanks!
Moz Pro | | onlineexpression
Karl0 -
Issue in number of pages crawled
i wanted to figure out how our friend Roger Bot works. On the first crawl of one of my large sites, the number of pages crawled stopped at 10000 (due to the restriction on the pro account). However after a few weeks, the number of pages crawled went down to about 5500. This number seemed to be a more accurate count of the pages on our site. Today, it seems that Roger Bot has completed another crawl and the number is up to 10000 again. I know there has been no downtime on our site, and the items that we fixed on our site did not reduce or increase the number of pages we had. Just making sure there are no known issues with Roger Bot before I look deeper into our site to see if there is an issue. Thanks!
Moz Pro | | cchhita0 -
SEOMOZ Competitive Domain Analysis
Hi, I am just trying to get to the bottom of why I am seeing such wildly different figures in the SEOMOZ link analysis. The SEO company that we use is saying that we have around 331 external links to our website, when I check on Yahoo site explorer, it returns 264 external links, but in my SEOMOZ campaign manager, it says I have 4 external links? I fully appreciate that none of these methods are going to be wholly accurate, and are going to produce different results, but I can't understand why the SEOMOZ results are so vastly different. I also know for a fact there are a number of Followed links that I have added to a couple of websites, that Google has now indexed (as they appear in search results), but don't appear in the campaign manager results. If SEOMOZ use a different method to assess an external link, then no problem - my main concern here is that as a Newbie I am doing something wrong within the SEOMOZ tools, and don't want to take action based on results that may be incorrect because I have set things up incorrectly? Any help is greatly appreciated. Many thanks, Gareth
Moz Pro | | gdavies090319770