Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Adjusting SEOmoz Crawling Speed

Moz Pro

868

cappuccino89 last edited by

How do you adjust the SEOmoz crawling speed?

SEOmoz tried to crawl 10,000 pages in 3 hours and crashed our MySQL server.
1 Reply Last reply
Reply Quote 1
kenneth_martin last edited by

Hello,

You can add a crawl delay to your robots.txt file.

It would look something like this:

User-Agent: rogerbot
Crawl-Delay: 5

(time is in seconds between page requests)

Hope that helps!

Kenny
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

SEOMoz API not working for Scrapebox

I want to import SEOMoz data to list of URLs I have using scrapbox. I added in my credentials according to the API but am getting error 401 as the status of all my links. Any idea why and what I should be doing?
Moz Pro | | theLotter

0
Have a Campaign, but only states 1 page has been crawled by SEOmoz bots. What needs to be done to have all the pages crawled?

We have a campaign running for a client in SEOmoz and only 1 page has been crawled per SEOmoz' data. There are many pages in the site and a new blog with more and more articles posted each month, yet Moz is not crawling anything, aside from maybe the Home page. The odd thing is, Moz is reporting more data on all the other inner pages though for errors, duplicate content, etc... What should we do so all the pages get crawled by Moz? I don't want to delete and start over as we followed all the steps properly when setting up. Thank you for any tips here.
Moz Pro | | WhiteboardCreations

0
Difference between SEOmoz Pro and Mozscape API

Hi, I probably put this question in a wrong category, so I'd like to ask again. I am new in SEOmoz and writing this from Japan.
I have couple of questions and really appriciate if you can help me to understand SEOmoz. better. First of all, what is the difference between SEOmoz Pro and Mozscape API? I signed up for 30days free trial and now continue being $99 PROmember.
Is that correct that rate limit is one request every 2 seconds with PRO? and also want to know about free API that I found after signed up for PRO.
Is there any restrictions on free API except for rate limit is one request every 10 seconds? The main reason I signed up for SEOmoz was to use the API link metrics, and because of the language I got really confused now.
Moz Pro | | flaminGoGo

0
Question regarding the SEOmoz Report Card

A report card is directed towards one keyword. What if one is to target two keywords on one page? For example, if I want to target SEO in Miami, Miami SEO, and SEO Miami it would be impossible to get an "A" on the report card for all three. What is the best way to target two-three keys on one page? Lets say these keywords are equally beneficial to the site, and we do not want to have to use hierarchy for one key over the other. Would it be wise to build a page for each? I feel like that would be redundant to the user. It says I need the keyword 4 times in the body so I should have 4 "SEO Miami" and 4 "Miami SEO". High importance factors. What makes them high importance? The keyword should be in the beginning of the title. How does this make sense as a high importance factor. Many titles are going to sound spammy if this is how they are written. Lets use Miami SEO again as an example: Miami SEO from [company name] bla bla bla sounds like spam. Learn about SEO in Miami from [company name] sound much better and is probably more likely to get clicked on. How important are these High Importance factors? I was always under the impression that we build the site for the user however the report card changes that perspective.
Moz Pro | | SEODinosaur

0
Pages Crawled: 250 | Limit: 250

One of my campaigns says: Pages Crawled: 250 | Limit: 250 Is this because it's new and the limit will go up to 10,000 after the crawl is complete? I have a pro account, 4 other campaigns running and should be allowed 50,000 pages in total
Moz Pro | | MirandaP

0
Can you change crawl day of week?

Can I somehow sync the day of the week for each of my campaigns' crawls, so that all campaigns are updated on the same day?
Moz Pro | | ATShock

0
SEOmoz crawl diagnostics report - what are the duplicate pages urls?

I just see the number of duplicates but not what the urls of the duplicates are? I don't see it in the export either, but maybe I'm missing it Cheers S
Moz Pro | | firstconversion

0
What's Your SEOmoz Favorite?

Obviously, SEOmoz is full of awesomeness. But I'm curious as to what sections, tools, etc. of SEOmoz are the users' favorites? For me, it's a close race between the blog, Linkscape/Open Site Explorer, and the search overlay tool on the Mozbar. What are your favorites?
Moz Pro | | Gyi

0