Adjusting SEOmoz Crawling Speed
-
How do you adjust the SEOmoz crawling speed?
SEOmoz tried to crawl 10,000 pages in 3 hours and crashed our MySQL server.
-
Hello,
You can add a crawl delay to your robots.txt file.
It would look something like this:
User-Agent: rogerbot
Crawl-Delay: 5(time is in seconds between page requests)
Hope that helps!
Kenny
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Seomoz crawl: 4XX (Client Error) How to find were the error are?
I got eight 404 errors with the Seomoz crawl, but the report does not says where the 404 page is linked from (like it does for dup content), or I'm I missing something? Thanks
Moz Pro | | PaddyDisplays0 -
Why SEOmoz bot consider these as duplicate pages?
Hello here, SEOmoz bot has recently marked the following two pages as duplicate: http://www.virtualsheetmusic.com/score/PatrickCollectionFlPf.html?tab=mp3 http://www.virtualsheetmusic.com/score/PatrickCollectionFlPf.html?tab=pdf I don't personally see how these pages can be considered duplicate since their content is quite different. Thoughts??!!
Moz Pro | | fablau0 -
Duplicate page titles in SEOMoz
My on page reports are showing a good number of duplicate title tags, but they are all because of a url tracking parameter that tells us which link the visitor clicked on. For example, http://www.example.com/example-product.htm?ref=navside and http://www.example.com/example-product.htm are the same page, but are treated as to different urls in SEOMoz. This is creating "fake" number of duplicate page titles in my reports. This has not been a problem with Google, but SEOMoz is treating it like this and it's confusing my data. Is there a way to specify this as a url parameter in the Moz software? Or does anybody have another suggestion? Should I specify this in GWT and BWT?
Moz Pro | | InetAll0 -
Crawl Diagnostics : Problem of display in Excell.
Hi Mozers, I've just finished watching the Crawl Diagnostics Webinar and when I try to export one of my campaign into the CSV format, I've a display problem into Microsoft Excell. Every headtitles are into the "A" column so, I can't do anything with that : I can't organize the data,... It's totally unreadable. What can I do? Thank you for yours answers. Jonathan
Moz Pro | | JonathanLeplang0 -
help with the inbound links side of seomoz
Hi Can somebody help with the inbound links tool of seomoz, can you point me in the direction of how it works and the best practices to get the most from it. I know i have a lot more inbound links to the site in my campaign but its shows for example 114 links but when i click on show more it only shows 3 . am i doing something wrong? Any help and advice from how people use it , what for and what is best practice thanks.
Moz Pro | | Bristolweb0 -
About NOFOLLOW tag for SEOmoz analysis
Hi all, Another issue while trying to resolve all the duplicate content SEOmoz reports to me. May be some of you guys can help: I have a dynamic error page on our website, generated in case of error, that can happen on many urls. Of course that one should not be indexed. I added the following tag on the HEADER: name="robots" content="NOODP,NOINDEX,NOFOLLOW" /> To me this should prevent from having this page indexed, but also from having this page reported by SEOmoz analyzer as duplicate content. Any hints?
Moz Pro | | nuxeo0 -
Why is the SEOmoz crawler crawling the old version of our website?
Hello, I'm a new SEOmoz member. On Dec. 2nd, after completely redesigning our website, we migrated to a new hosting company by switching our DNS to the new server. The vast majority of the URLs have changed and we configured redirects of the old URLs to the new ones. Although, this task is not completed yet. After the migration, I created an account on SEOmoz to be able to track our progress and find the issues to fix to optimize our SEO. For some reason, in the SEOmoz reports it is the old URLs that show up. Unless the crawler does not actually crawl the pages and only uses the indexed pages to generate its report, I don't understand how could this possible. Anyone has a clue? When will the new URLs be indexed by SEOmoz and the major search engines? Thanks for your help!
Moz Pro | | Gestisoft-Qc0 -
How to set up SEOMOZ to track multilanguage sites?
Hi, I am managing a site in 5 different languages/regions. Our language structure is as follows: www.domain.com/en-us/ - For US www.domain.com/en-uk/ - For UK www.domain.com/fr/ - For France ... However, SEOMOZ does not allow to set up a domain with "/" as a campaign. How can I do it? Thanks
Moz Pro | | hockerty1