Slowing down SEOmoz Crawl Rate
-
Is there a way to slow down SEOmoz crawl rate? My site is pretty huge and I'm getting 10k pages crawled every week, which is great. However I sometimes get multiple page requests in one second which slows down my site a bit.
If this feature exists I couldn't find it, if it doesn't, it's a great idea to have, in a similar way to how Googlebot do it.
Thanks.
-
Thank you for the reply Megan, just what I was looking for.
-
Hi corwin,
This is Megan from the SEOmoz Help Team. I'm sorry if roger is being a bit too aggressive. We built our crawler to obey robots.txt crawl-delay directives. If you ever need to speed him down (though he's usually well-behaved), just add a crawl delay directive to your robots.txt file like this:
User-agent: rogerbotCrawl-delay: 1
Here's a good article that explains more about this technique: http://tools.seobook.com/robots-txt/
I hope this helps!
Cheers
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEOmoz Not Picking Up Links
I'm finding that SEOmoz often doesn't pick up on many of the links my client's websites are attaining. The links do appear in search engine and often have a high domain authority (often PR links, which I why I follow up to see if they've been picked up). I do tend to see many of the links appear in webmaster tools, so it's a little frustrating that they don't appear, even after months of pages being live. Thoughts?
Moz Pro | | SebastianDyer0 -
SEOMoz Langage ?
Hello, Is there any way to change the lagage of the interface ? Especially for reporting tool ? Thanks in advance for your answer, Alex
Moz Pro | | ACORED0 -
What could be the reason that seomoz only shows crawl results for my homepage?
Hi there I am running three campaings for three different sites. The first site crawl is successful with a ful report. However the other two only shows results for the homepage, i.e. only a single page crawled by the mozbot. What could be the reason for this? Thanks, Gerrie
Moz Pro | | marketingmen0 -
SEOMoz Q&A having some issues?
Apologies if this has been asked. I was browsing Q&A and got a Ruby error page, reloaded and everything was fine. However a bunch of the topics are kind of messed up, and it looks like a chunk of them are missing now. For example my 'questions I've answered' section of My Q&A only has the newest question and the oldest question, and when I sort the SEOMoz Tools category by newest I get one recent post and then the next one is from November of last year. You guys having database problems?
Moz Pro | | icecarats0 -
Crawl frequency
What is the frequency of crawl. My crawl diagnostics shows data from over a month ago
Moz Pro | | deBreezeInteractive0 -
SEOmoz Pro dashboard report question
Hi there, SEOMoz dashbard report shows 'URLs Receiving Entrances Via Search' - Is it counting only organic searches or all searches including PPC? Thanks, Gemma
Moz Pro | | RBA0 -
An error in the SeoMoz On page note?
Hello folks, Whenever I go the OnPage link in SeoMoz some of my links show a F ranking note. And when I click in one of them to see the detail of the page rank, it shows me as an A ranking note. Do you have seen the same problem? Which note shall I rely on? Thanks!!
Moz Pro | | jgomes0