Unsolved The Moz.com bot is overloading my server
-
. How to solve it?
-
maybe crawl delay will help.
-
@paulavervo
Hi,
We do! The best way to chat with us is via our contact form or direct email. We also have chat within Moz Pro.
Please contact us via help@moz.com or https://moz.com/help/contact
We will be happy to help.
Cheers,
Kerry. -
very nice brother i like it very good keep it up !
-
very nice !
-
does the moz team even monitor this forum?
-
If the Moz.com bot is overloading your server, there are several steps you can take to manage and mitigate the issue effectively. First, you can adjust the crawl rate in your
robots.txt
file by specifying a crawl delay for the Moz bot using directives likeUser-agent: rogerbot
andUser-agent: dotbot
, followed byCrawl-delay: 10
to make the bot wait 10 seconds between requests. If this does not suffice, you can temporarily block the bot by disallowing it in yourrobots.txt
file. Additionally, it's a good idea to contact Moz’s support team to explain the issue, as they may offer solutions to adjust the crawl rate for your site. Implementing server-side rate limiting is another effective strategy. For Apache servers, you can add rules in your.htaccess
file to return a 429 Too Many Requests status code to the Moz bots, while for Nginx servers, you can set up rate limiting in your configuration file to control the number of requests per second from a single user or IP address. Monitoring your server’s performance and log files can help identify specific patterns or peak times, allowing you to fine-tune your settings. Furthermore, using a Content Delivery Network (CDN) can help distribute the load by caching content and serving it from multiple locations, reducing the direct impact on your server caused by crawlers. By taking these steps, you can manage the load from the Moz.com bot and maintain your server’s stability and responsiveness.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Rogerbot directives in robots.txt
I feel like I spend a lot of time setting false positives in my reports to ignore. Can I prevent Rogerbot from crawling pages I don't care about with robots.txt directives? For example., I have some page types with meta noindex and it reports these to me. Theoretically, I can block Rogerbot from these with a robots,txt directive and not have to deal with false positives.
Reporting & Analytics | | awilliams_kingston0 -
Unsolved 503 Service Unavailable (temporary?) Rogerbot takes a break
A lot of my Moz duties seem to be setting hundreds of issues to ignore because my site was getting crawled while under maintenance. Why can't Rogerbot take a break after running into a few of these and then try again later? Is there an official code for Temporary Service Unavailability that can smart bots pause crawls so that they are not wasting compute, bandwidth, crawl budget and my time?
Product Support | | awilliams_kingston0 -
Unsolved Why is'nt there an update yet!
0 -
Does increase in DA increase Moz Trust score?
I have checked many website which in my office. Have a confusion Does increase in DA increase Moz Trust score? I have a website https://emocrest.com/ which is new one and Moz Trust score as of now is 1 only. Well, most of the projects I'm handling in our office having 2,3,4 in the range. How this MT value is calculated and does it got any relation with the change in Domain Authoiry or the page authority of a website?
Moz Pro | | nazfazy1 -
Why is MOZ crawl is returning URLs with variable results showing Missing Meta Desc? Example: http://nw-naturals.net/?page_number_0=47
Can you help me dive down into my website guts to find out why the MOZ crawl is returning URLs with variable results? And saying this is missing a description when it's not really a page? Example: http://nw-naturals.net/?page_number_0=47. I've asked MOZ but it's a web development issue so they can't help me with it. Has anyone had an issue with this on their website? Thank you!
Moz Pro | | lewisdesign0 -
Open Site Explorer produces better results for www.50campfires.com than 50campfires.com
Our website url is 50campfires.com and www.50campfires.com redirects to 50campfires.com Look at the Open Site Explorer results for the two and www.50campfires.com has better results. Does google have an issue with this or may it be confusing them? We have redirects in place. How would I go about fixing this? More on what strategies would improve Page Authority and Page Rank. Thanks in Advance. 79JtBKA.png lrHwsOU.png
Moz Pro | | revonick0 -
Moz scores
Hi All, Could anyone please explain for me why the moz scores that show under SERPs in Google search show one set of values, but when i click on the SERP listing and go to that page, the Moz scores at the bottom of the tab the page is open in are completely different. I.E - SERP result Wireless Alarms <cite>www.wireless-alarms.net/</cite>Honeywell, Friedland, Response, Wireless house home intruder burglar alarms and wirefree doorbells chimes and accessories.<iframe class="SEOmoz-iframe" src="chrome-extension://eakacpaijcpapndcfffdgphdiccmpknp/html/serpbar.html#{" settings":{"dmr":false,"dmt":false,"mr":true,"mt":true,"serp-overlay":true,"toolbar-enabled":true,"subdomain-metrics":false,"serp-panel-open":true,"toolbar-position":"bottom"},"lsdata":{"feid":3,"fipl":3,"fmrp":2.0446037910058426,"fmrr":5.396862303226778e-12,"ftrp":1.8723654310590483,"ftrr":9.21604483751679e-12,"fuid":3,"pda":8.903939958502189,"peid":3,"pid":3,"pmrp":1.6026230610187469,"pmrr":1.3566326316606153e-10,"ptrp":1.025503845909542,"ptrr":1.4972548621304158e-11,"puid":3,"ueid":2,"uemrp":2.577887966559377,"uemrr":5.40110633420739e-13,"uid":2,"uipl":2,"ujid":2,"umrp":2.577887966559377,"umrr":5.40110633420739e-13,"upa":23.634692786039714,"utrp":4.9331532541404854,"utrr":1.9684091945588755e-16},"user":{"level":"pro","options":{},"proenabled":true,"aclurl":"http:="" moz.com="" users="" level?src="mozbar","loginURL":"https://moz.com/login","logoutURL":"https://moz.com/logout","apiRequest":"url-metrics","cols":"133146078688","expired":false,"accessId":"pro-RGFkZHlTbXVyZg%3D%3D","expires":1374925378,"signature":"tBSBXpLbBmk073ofsfBwnilmODw%3D","displayName":"DaddySmurf"},"serp":4,"host":"www.wireless-alarms.net"}"" scrolling="no"></iframe> Page result <label>Page:</label><label id="Page Authority" class="key lsdata">PA:</label> <label id="SEOmoz-data-PA" class="val lsdata">27</label><label id="mozrank" class="key lsdata" title="MozRank">mR:</label> <label id="SEOmoz-data-mR" class="val lsdata">5.30</label><label id="moztrust" class="key pro-only lsdata" title="MozTrust">mT:</label> <label id="SEOmoz-data-mT" class="val pro-only lsdata">5.61</label><label id="SEOmoz-data-uid">1,950</label> links from
Moz Pro | | DaddySmurf
<label id="SEOmoz-data-uipl">2</label> Root Domains<label>Root Domain:</label><label id="dom-pageauthority" class="key lsdata" title="Domain Authority">DA:</label> <label id="SEOmoz-data-DA" class="val lsdata">24</label><label id="SEOmoz-data-puid">100,262</label> links from
<label id="SEOmoz-data-pid">3</label> Domains<label>Subdomain:</label><label id="SEOmoz-data-fuid">19,953</label> links from
<label id="SEOmoz-data-fipl">3</label> DomainsThank you.Si0