Unsolved The Moz.com bot is overloading my server
-
. How to solve it?
-
maybe crawl delay will help.
-
@paulavervo
Hi,
We do! The best way to chat with us is via our contact form or direct email. We also have chat within Moz Pro.
Please contact us via help@moz.com or https://moz.com/help/contact
We will be happy to help.
Cheers,
Kerry. -
very nice brother i like it very good keep it up !
-
very nice !
-
does the moz team even monitor this forum?
-
If the Moz.com bot is overloading your server, there are several steps you can take to manage and mitigate the issue effectively. First, you can adjust the crawl rate in your
robots.txt
file by specifying a crawl delay for the Moz bot using directives likeUser-agent: rogerbot
andUser-agent: dotbot
, followed byCrawl-delay: 10
to make the bot wait 10 seconds between requests. If this does not suffice, you can temporarily block the bot by disallowing it in yourrobots.txt
file. Additionally, it's a good idea to contact Moz’s support team to explain the issue, as they may offer solutions to adjust the crawl rate for your site. Implementing server-side rate limiting is another effective strategy. For Apache servers, you can add rules in your.htaccess
file to return a 429 Too Many Requests status code to the Moz bots, while for Nginx servers, you can set up rate limiting in your configuration file to control the number of requests per second from a single user or IP address. Monitoring your server’s performance and log files can help identify specific patterns or peak times, allowing you to fine-tune your settings. Furthermore, using a Content Delivery Network (CDN) can help distribute the load by caching content and serving it from multiple locations, reducing the direct impact on your server caused by crawlers. By taking these steps, you can manage the load from the Moz.com bot and maintain your server’s stability and responsiveness.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved 503 Service Unavailable (temporary?) Rogerbot takes a break
A lot of my Moz duties seem to be setting hundreds of issues to ignore because my site was getting crawled while under maintenance. Why can't Rogerbot take a break after running into a few of these and then try again later? Is there an official code for Temporary Service Unavailability that can smart bots pause crawls so that they are not wasting compute, bandwidth, crawl budget and my time?
Product Support | | awilliams_kingston0 -
Unsolved Why does Moz Crawler start with HTTP//1.1 version??
We have run Moz Pro's Site Crawl for example-domain.com Why has Moz's crawler decided to site http://www.example-domain.com (ie the non-secure version) as zero crawl depth page and report the actually live https://www.example-domain.com (ie secure HTTP2 version) as a crawl depth of 2? Surely the main live page should be the first crawled and reported as crawl depth = 0?
Moz Pro | | AKCAC0 -
Unsolved Moz can't crawl my site
Moz is being blocked from crawling the following site - https://www.cleanchain.com. When looking at Robot.txt, the following is disallowing access but don't know whether this is preventing Moz from crawling too? User-agent: *
Moz Pro | | danhart2020
Disallow: /adeci/
Disallow: /core/
Disallow: /connectors/
Disallow: /assets/components/ Could something else be preventing the crawl?0 -
Unsolved MOZ Crawler Stalled
MOZ crawl has been waiting since April 2nd. Whats the deal? I am not able to "recrawl", so i have been stuck in limbo.
Product Support | | WebMarkets0 -
I see some links on Youtube.com that look like they are "Do-follow" links am I wrong?
Hi, I have some videos on YouTube.com and I notice many of the links show as "Do-Follow" per the Moz Bar. I have a feeling they are not actually "Dofollow" links because they never show up in Moz Pro, SemRush or any of my other tools. Can some of you people with greater knowledge of YouTube please clarify why the links look like "Do Follow". I am specifically thinking of my own backlinks on the videos and also the links in the "Comments". Best Regards, Bill Dalessi
Moz Pro | | Dalessi0 -
A blank moz bar
Hey MOZ peeps, iv come across something strange. The website http://abundantiaentertainment.com/ displays a "blank" MOZ bar ?! Can anyone help me understand this Thank you in advance
Moz Pro | | Silverwaring0 -
Recommended 3rd party Moz compatible dashboard
Hi, I'd like to use a 3rd party dashboard to monitor all marketing activities in one place (social / seo from moz/ remarketing / email / content / google analytics etc.) - any recommendations? Thanks Louisa
Moz Pro | | charliedavi0 -
Authority from Linking Root Domains: youtube.com / wikipedia.org / adobe.com
Hi there, Presently doing competitor analysis and note two competitors who have a way higher 'moz domain authority' than my client. Using moz tools I notice their top 5 linking root domains all have a score of 100. Refer to screen shot. Of note, both list youtube.com and _wikipedia.org. _ Similarly, my client's domain is ALSO linked from their user profile on youtube.com. They also have a published wiki page with their URL linked. BUT, youtube.com or wikipedia.org are not listed in their "top 5 linking root domains". Their highest scoring linking root domain is prweb.com - with a score of 97. If my client has links on these top domains why would they not be listed in my client's top five domains list like they are listed in their competitors top five? Researching for reasons I came across this old post (2009) here - http://moz.com/blog/followed-links-from-four-unexpected-sources - and wonder if the competitor's links are 'followed' links - even though all resources suggest wiki and youtube are definitely 'no follow' links? Other interesting "Top 5" domains that are listed for my competitors as top "linking root domains" are microsoft.com, adobe.com and europa.eu - again, refer to screenshot. Questions are IF these top linking root domains are in fact 'followed' links/valuable links and help with domain authority scores calculated by the moz tool then 1) HOW do I get these links to show/provide the same value? AND 2) How are my competitors, who are simply travel products, getting links from top domains like adobe.com? I do hope all the above makes sense and that I'm using/interpreting the moz comparative tool correctly! Cheers iGe864i.jpg?1
Moz Pro | | catherineh0