What's the best way to eliminate "429 : Received HTTP status 429" errors?
-
My company website is built on WordPress. It receives very few crawl errors, but it do regularly receive a few (typically 1-2 per crawl) "429 : Received HTTP status 429" errors through Moz.
Based on my research, my understand is that my server is essentially telling Moz to cool it with the requests. That means it could be doing the same for search engines' bots and even visitors, right? This creates two questions for me, which I would greatly appreciate your help with:
-
Are "429 : Received HTTP status 429" errors harmful for my SEO? I imagine the answer is "yes" because Moz flags them as high priority issues in my crawl report.
-
What can I do to eliminate "429 : Received HTTP status 429" errors?
Any insight you can offer is greatly appreciated!
Thanks,
Ryan -
-
I have a customer that is using GoDaddy website hosting (at least according to BuiltWith) and I'm experiencing this same issue.
Any updates on this experiment from user rsigg? I'd love to know if I can remove this from my customer's robots file...
FWIW, Netrepid is a hosting provider for colocation, infrastructure and applications (website hosting being considered an application) and we would never force a crawl delay on a Wordpress install!
Not hating on the other hosting service providers... #justsayin
-
I am also on the same hosting and they have not been able to help with the 429. I have now started getting 429 errors when I attempt to login. Definitely something wrong with wp premium hosting.
-
Interesting. I look forward to hearing your results, as my robots.txt file is also set to:
Crawl-delay: 1.
-
We host on Media Temple's Premium WordPress hosting (which I do not recommend, but that's another post for another place), and the techs there told me that it could be an issue with the robots.txt file:
"The issue may be with the settings in the robots.txt file. It looks fine to me but the "Crawl-delay" line might be causing issues. I understand. For the most part, crawlers tend to use robots.txt to determine how to crawl your site, so you may want to see if Moz requires some special settings in there to work correctly."
Ours is set to:
Crawl-delay: 1
I haven't tried changing these values yet in our file, but may experiment with this very soon. If I get results, I'll post back here as well as start a new forum thread.
-
Chase,
They ran a bunch of internal diagnostic tools on my site, and were unable to replicate the 429 errors. They ended up telling me exactly what they told you. I haven't noticed any issues with my site's rankings, or any flags in Webmaster Tools, so it looks like they are right so far. I just hate logging into Moz and seeing all those crawl errors!
-
What'd they say Ryan? Having the same issue and just contacted Godaddy who told me that basically Moz's software is pinging my client's server too frequently so Godaddy is temporarily blocking their IP. They said it's not a concern though as they would never block Google from pinging/indexing the site.
-
Many thanks - I will contact them now!
-
Contact your host and ask let them know about the errors. More than likely they have mod_sec enabled to limit request rates. Ask them to up the limit that you are getting 429 errors from crawlers and you do not want them.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I have 702 'No-Index' warnings. Is this bad?
Moz has giving me 702 'No-Indexed Meta-descriptions' warnings. My page has quite a bit of product pages as it is a commercial chemical company which sells cleaning products for restaurants, hospitals, etc. Im wondering if this is effecting my site negatively?
Moz Pro | | ACSmt0 -
Could my Crawl Error Report be wrong?
HI there, I am using Yoast SEO plugin on a wordpress website. I am currently showing 70 Med priority crawl errors 'missing meta description' on my Moz pro account. This number of missing meta descriptions has increased over the last 6 weeks. But every single page / post / tag / category has both the title and meta description completed via Yoast. I requested a google bot to crawl the site a few weeks ago as thought it perhaps wasn't been crawled and updated. Any idea what the issue might be? Could the Moz report be incorrect for some reason? Or could something be blocking Moz / Google from seeing the Yoast plugin?
Moz Pro | | skehoe0 -
UK Google Results .com V's .co.uk
In the UK lots of people use google.com rather than google.co.uk. See: http://weblogs.hitwise.com/robin-goad/2008/04/how_popular_is_googles_pages_from__the_uk_search_option.html The above data is a bit out of date but still I know lots of UK people who search on google.com. When I depersonalise the results and search there is a difference between what I see on google.com V's what I see on google.co.uk. Indeed one of the companies I work for ranks ok on google.co.uk, but not so well on google.com Questions: How would I best track ranking for UK based users who use google.com AND/OR google.co.uk using the SEOmoz tools (or other tools if appropriate)? I note that in "Google Webmaster Tools" you can change the setings to "Target users in United Kingdom". Given the fact that UK users use both google.com AND/OR google.co.uk would it be better to change this to unlisted or deselect the "Target users in" option? Thanks in advance. Justin
Moz Pro | | GrouchyKids0 -
SEOmoz bot and "noindex"
As a recent newbie to SEOmoz, I've been implementing some suggestions and doing a general tidy up. I removed URL's from our robots txt, and rolled out instead the noindex meta tag to pages we don't want indexed. But surprised to see issues that are now flagged from the last crawl by the moz bot from pages that have this meta tag? Does the SEOmoz bot not ignore this tag? Just want to make sure I've implemented it correctly, so the google bot does ignore it. Meta tag syntax is and is placed below the title tag. cheers Steve
Moz Pro | | sjr4x40 -
Keyword Difficulty Tool: Error
Hi - is anyone else getting an error using the Keyword Difficulty tool? I'm getting "ERROR: There was a transient error with your request. Please try again."
Moz Pro | | ErikDster0 -
Site is penalized yet Google WTC and SEOMOZ says it's ok!
My first question on this community.. A high traffic site of mine has been severely hit for over two months. I took a look at GWTC and noticed a lot of suggestions on the HTML suggestions tab, where I believed that one or a combination of these errors/notices might been causing the penalty. At the moment of typing GWTC shows no HTML suggestions and minor meaningless crawl errors. Checking on SEOMOZ reported issues, I see zero errors except for a high number of duplicate page titles resulting from tag pages being accessible in the same tag title in paginations (tag.html?page=x) which can be fixed easily by using rel=canonical but I just know it can't cause such penalty. For reported warnings, site has a decent amount of title element being too long and missing meta tag description tags. I have been very active in the past with 301 inner pages of the site to make pages URLs as unique as possible and results were great, but that was long time ago before taking the hit. The penalty the site suffers from is zero rankings for newly added contents and old contents as well. It has also lost all rankings for its money terms. Site is not banned and still ranking top for its name which is a good thing. I am not sure where else I should be looking and will appreciate any good advices. Thanks!
Moz Pro | | HossamHossny0 -
Why would PA be 1 (0 links from 0 root domains) if it's linked to internally?
Question just about said it all: I've seen a number of pages on sites that have a PA of 1 (with the metrics being 0 links from 0 root domains) when I can see on the site that it is linked to internally - from the main nav (which is CSS, not Javascript) and also from the footer, if not other places. Why would this be? Update: upon looking further at the site, it appears that there's some kind of redirect going on, where the page linked to from the nav actually redirects to the real page. Would that eliminate PA, even if it's a 301? And additionally, is whatever is causing this lack of PA a reflection of how Google would relate to the page? Thanks, Aviva
Moz Pro | | debi_zyx0 -
Is there such thing as a site free from errors?
Or is this a given? I am new to SEO and SEOmoz. One of my campaigns is completley free of errors...the others are a work in progress. Now I realize that SEO is never done, but can a site actually be free of errors? If so... I just gave myself a pat on the back.
Moz Pro | | AtoZion0