What's the best way to eliminate "429 : Received HTTP status 429" errors?
-
My company website is built on WordPress. It receives very few crawl errors, but it do regularly receive a few (typically 1-2 per crawl) "429 : Received HTTP status 429" errors through Moz.
Based on my research, my understand is that my server is essentially telling Moz to cool it with the requests. That means it could be doing the same for search engines' bots and even visitors, right? This creates two questions for me, which I would greatly appreciate your help with:
-
Are "429 : Received HTTP status 429" errors harmful for my SEO? I imagine the answer is "yes" because Moz flags them as high priority issues in my crawl report.
-
What can I do to eliminate "429 : Received HTTP status 429" errors?
Any insight you can offer is greatly appreciated!
Thanks,
Ryan -
-
I have a customer that is using GoDaddy website hosting (at least according to BuiltWith) and I'm experiencing this same issue.
Any updates on this experiment from user rsigg? I'd love to know if I can remove this from my customer's robots file...
FWIW, Netrepid is a hosting provider for colocation, infrastructure and applications (website hosting being considered an application) and we would never force a crawl delay on a Wordpress install!
Not hating on the other hosting service providers... #justsayin
-
I am also on the same hosting and they have not been able to help with the 429. I have now started getting 429 errors when I attempt to login. Definitely something wrong with wp premium hosting.
-
Interesting. I look forward to hearing your results, as my robots.txt file is also set to:
Crawl-delay: 1.
-
We host on Media Temple's Premium WordPress hosting (which I do not recommend, but that's another post for another place), and the techs there told me that it could be an issue with the robots.txt file:
"The issue may be with the settings in the robots.txt file. It looks fine to me but the "Crawl-delay" line might be causing issues. I understand. For the most part, crawlers tend to use robots.txt to determine how to crawl your site, so you may want to see if Moz requires some special settings in there to work correctly."
Ours is set to:
Crawl-delay: 1
I haven't tried changing these values yet in our file, but may experiment with this very soon. If I get results, I'll post back here as well as start a new forum thread.
-
Chase,
They ran a bunch of internal diagnostic tools on my site, and were unable to replicate the 429 errors. They ended up telling me exactly what they told you. I haven't noticed any issues with my site's rankings, or any flags in Webmaster Tools, so it looks like they are right so far. I just hate logging into Moz and seeing all those crawl errors!
-
What'd they say Ryan? Having the same issue and just contacted Godaddy who told me that basically Moz's software is pinging my client's server too frequently so Godaddy is temporarily blocking their IP. They said it's not a concern though as they would never block Google from pinging/indexing the site.
-
Many thanks - I will contact them now!
-
Contact your host and ask let them know about the errors. More than likely they have mod_sec enabled to limit request rates. Ask them to up the limit that you are getting 429 errors from crawlers and you do not want them.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
302 Status Code for .com but 200 Status Code for .com/
I ran a moz crawl report for one of my sites. For mywebsite.com I'm getting a 302 HTTP Status Code but for mywebsite.com/ I'm getting a 200 HTTP Status Code. mywebsite.com -Title tag "blank" - Meta Description "blank" - HTTP Status Code "302" - Referrer "blank" - Content Type Header "blank" - 301/302 target "/" - Rel-Canonical Target "blank" mywebsite.com/ - Title tag "correct tag" - Meta Description "correct tag" - HTTP Status Code "200" - Referrer "mysite.com" - Content Type Header "text/html" - 301/302 target "blank" - Rel-Canonical Target "mysite.com/" Will this cause any issues with the search engines? Is there a way to fix this? I'm on wordpress and use Yoast SEO premium plugin. I have sites ranking in many different niches and have done similar white hat SEO for all of my others. They are ranking in competitive niches but for some reason this site is in the 200-300s for basically all keywords in not a very competitive niche. It has excellent content, links and no penalties. I'm trying to find anything that could be the cause of this. Thanks! <colgroup><col width="64"></colgroup>
Moz Pro | | cccreationsinc
| Rel-Canonical Target |0 -
Rank Checker Tool Status
Hi I take it the Rank Checker tool is not working right yet as I'm getting bizarre ranks for some of my keywords. Some stating 50th when I know we're on the first page for instance, even using GA > search engine optimisation the averages for this term for instance is 11th across all countries. If this is the case and its still being redeveloped I'll just skip the tool for now Cheers David
Moz Pro | | David-E-Carey0 -
Can I calculate "Keyword Difficulty" metric using Mozscape API data?
We already have a web application that pulls certain metrics about websites using the Mozscape API, but we are wanting to extend the usefulness of this application to enable users of the app to pull "Keyword Difficulty" metrics in bulk, instead of one at a time (or 5 at a time). I wouldn't mind the 5 at a time limitation if we could just automate the API calls and let the tool pull data for 50 or so keywords without user-interaction. I know that it's a "formula", but I don't know what SEOMoz uses for it's formula. Has anyone figured out a way to calculate this, based on the Mozscape API data? Has anyone ever tried to reverse engineer this metric?
Moz Pro | | brchap0 -
Here's a hard one! Why isn't my profile picture displaying as an avatar iin forums and blogs?
My profile pic is updated and displays on my profile but I still have the default avatar beside all forum and blog posts... This is life or death... I must know the answer. 😉 Thanks guys.
Moz Pro | | Anthony_NorthSEO0 -
In OSE "Followed Linking Root Domains" = "links from homepages"
In OSE's, "Followed Linking Root Domains" are defined as "The number of root domains that have at least one followed link to a page or domain." Does this mean that if one of my competitors has, let's say, 1000 followed linking root domains, they have a link pointing to them on the homepage of 1000 other sites? Thanks for your help!
Moz Pro | | gerardoH0 -
20000 site errors and 10000 pages crawled.
I have recently built an e-commerce website for the company I work at. Its built on opencart. Say for example we have a chair for sale. The url will be: www.domain.com/best-offers/cool-chair Thats fine, seomoz is crawling them all fine and reporting any errors under them url great. On each product listing we have several options and zoom options (allows the user to zoom in to the image to get a more detailed look). When a different zoom type is selected it adds on to the url, so for example: www.domain.com/best-offers/cool-chair?zoom=1 and there are 3 different zoom types. So effectively its taking for urls as different when in fact they are all one url. and Seomoz has interpreted it this way, and crawled 10000 pages(it thinks exist because of this) and thrown up 20000 errors. Does anyone have any idea how to solve this?
Moz Pro | | CompleteOffice0 -
Why am I getting duplicate content errors on same page?
In the SEOmoz tools I am getting multiple errors for duplicate page content and duplicate page titles for one section on my site. When I check to see which page has the duplicate title/content the url listed is exactly the same. All sections are set up the same, so any ideas on why I would be getting duplication errors in just this one section and why they would say the errors are on the same page (when I only have one copy uploaded on the server)?
Moz Pro | | CIEEwebTeam0 -
Is there any way to change the domain in a campaign here?
When I started the campaign I used site.com but now I have an external blog at blog.site.com and is hosted on tumblr. This site gets lots of errors showing on the campaign because insists on following no-follows. If I change the campaign to www.site.com then blog.site.com will not be part of the campaign. It does say in the help that it will NOT track sub-domains, but it does and hence my problem. So, I am not sure if the rest of the info there is old. 'Does the Web App track the subdomains of my campaign’s domain?
Moz Pro | | oznappies
A. Unfortunately we do not currently track subdomains as a part of the
domain you enter. Instead, you must create another campaign slot for any
subdomains you wish to track.' Any ideas on where or how to do this?0