What's the best way to eliminate "429 : Received HTTP status 429" errors?
-
My company website is built on WordPress. It receives very few crawl errors, but it do regularly receive a few (typically 1-2 per crawl) "429 : Received HTTP status 429" errors through Moz.
Based on my research, my understand is that my server is essentially telling Moz to cool it with the requests. That means it could be doing the same for search engines' bots and even visitors, right? This creates two questions for me, which I would greatly appreciate your help with:
-
Are "429 : Received HTTP status 429" errors harmful for my SEO? I imagine the answer is "yes" because Moz flags them as high priority issues in my crawl report.
-
What can I do to eliminate "429 : Received HTTP status 429" errors?
Any insight you can offer is greatly appreciated!
Thanks,
Ryan -
-
I have a customer that is using GoDaddy website hosting (at least according to BuiltWith) and I'm experiencing this same issue.
Any updates on this experiment from user rsigg? I'd love to know if I can remove this from my customer's robots file...
FWIW, Netrepid is a hosting provider for colocation, infrastructure and applications (website hosting being considered an application) and we would never force a crawl delay on a Wordpress install!
Not hating on the other hosting service providers... #justsayin
-
I am also on the same hosting and they have not been able to help with the 429. I have now started getting 429 errors when I attempt to login. Definitely something wrong with wp premium hosting.
-
Interesting. I look forward to hearing your results, as my robots.txt file is also set to:
Crawl-delay: 1.
-
We host on Media Temple's Premium WordPress hosting (which I do not recommend, but that's another post for another place), and the techs there told me that it could be an issue with the robots.txt file:
"The issue may be with the settings in the robots.txt file. It looks fine to me but the "Crawl-delay" line might be causing issues. I understand. For the most part, crawlers tend to use robots.txt to determine how to crawl your site, so you may want to see if Moz requires some special settings in there to work correctly."
Ours is set to:
Crawl-delay: 1
I haven't tried changing these values yet in our file, but may experiment with this very soon. If I get results, I'll post back here as well as start a new forum thread.
-
Chase,
They ran a bunch of internal diagnostic tools on my site, and were unable to replicate the 429 errors. They ended up telling me exactly what they told you. I haven't noticed any issues with my site's rankings, or any flags in Webmaster Tools, so it looks like they are right so far. I just hate logging into Moz and seeing all those crawl errors!
-
What'd they say Ryan? Having the same issue and just contacted Godaddy who told me that basically Moz's software is pinging my client's server too frequently so Godaddy is temporarily blocking their IP. They said it's not a concern though as they would never block Google from pinging/indexing the site.
-
Many thanks - I will contact them now!
-
Contact your host and ask let them know about the errors. More than likely they have mod_sec enabled to limit request rates. Ask them to up the limit that you are getting 429 errors from crawlers and you do not want them.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Spammy inbound links: Don't Fix It If It's Not Broken?
Hi Moz community, Our website is nearing the end of a big redesign to be mobile-responsive. We decided to delay any major changes to text content so that if we do suffer a rankings drop upon launch, we'll have some ability to isolate the cause. In the meantime I'm analyzing our current SEO strengths and weaknesses. There is a huge discrepancy between our rankings and our inbound link profile. Specifically, we do great on most of our targeted keywords and in fact had a decent surge in recent months. But Link Profiler turned up hundreds of pages of inbound links from spammy domains, many of which don't even display a webpage when I click there. (shown in uploaded image) "Don't fix it if it's not broken" is conflicting with my natural repulsion to these sorts of referrals. Assuming we don't suffer a rankings drop from the redesign, how much of a priority should this be? There are too many and most are too spammy to contact the webmasters, so we'll need to do it through a Disavow. I couldn't even open the one at the top of the list because our business web proxy identified it as adult content. It seems like a common conception is that if Google hasn't penalized us for it yet, they will eventually. Are we talking about the algorithm just stumbling upon these links and hurting us or would this be something we would find in Manual Actions? (or both?) How long after the launch should we wait before attacking these bad links? Is there a certain spam score that you'd say is a threshold for "Yes, definitely get rid of it"? And when we do, should we Disavow domains one domain at a time to monitor any potential drops or all at once? (this seems kind of obvious but if the spam score and domain authority alone is enough of a signal that it won't hurt us, we'd rather get it done asap) How important is this compared to creating fresh new content on all the product pages? Each one will have new images as well as product reviews, but the product descriptions will be the same ones we've had up for years. I have new content written but it's delayed pending any fallout from the redesign. Thanks for any help with this! d1SB2JP.jpg
Moz Pro | | jcorbo0 -
Does google see the keyword "e cig" and "e-cig" as the same word? on MOZ it shows that they have a totally different amount of search quarries.
does google see the keyword "e cig" and "e-cig" as the same word? on MOZ it shows that they have a totally different amount of search quarries.
Moz Pro | | smokingllc0 -
Best Way to Seemlessly Use Moz and Google Analytics
Hi All, I have Moz Pro and I am beginning to explore Moz Analytics. I currently use Google Analytics for pretty much everything and I am almost at expert level at using the GA interface so I'm ready to learn a new tool and have a second source of data! My question is: what's the best way to get the most of Moz Analytics and GA congruently? Off hand I know that Moz's keyword, rank, and link data, including competitive insights, is more comprehensive than GA. But how else can I use Moz and get the most out of the Moz Analytics? Looking to really dive in and discuss how we can make use of both! Thanks - Vee
Moz Pro | | Symmetri0 -
Is there any way to get historical API data?
I'm looking for a way of requesting domain authority and some other stats via the API for several previous months. I noticed on here: http://apiwiki.seomoz.org/what-is-the-mozscape-api
Moz Pro | | ReallyEasyCart
It says "SEOmoz PRO members now have access to historical API linking data - read about this great new feature on our blog!" but i could not find this blog post anywhere and maybe I'm misreading this line anyway.0 -
Can I setup SEOmoz PRO's 3 organic traffic metrics within Google Analytics (as custom reports)?
I like the 3 organic traffic metrics in SEOmoz PRO: 1) Organic Search Visits
Moz Pro | | AndreVanKets
2) URLs Receiving Entrances Via Search
3) Non-Paid Keywords Sending Search Visits They rock - especially for a high-level view of SEO / Organic performance. Is it possible to setup these metrics as custom reports directly in Analytics? Does anyone have these reports setup? Or maybe there are similar custom reports that give a similar high-level view?0 -
How accurate is the MozBar "Followed Link Highlighter"
I'm currently doing some research on blogs to guest post on and I want to make sure they are providing DoFollow links out. are the green links displayed from the MozBar simply links that don't contain the HTML element "NOFOLLOW" or is it more in depth tan that? according to this article there are TONS of different ways that people can place a NOFOLLOW link on their site http://www.searchenginepeople.com/blog/identify-nofollow-juiceless-links-guide.html Thanks
Moz Pro | | SheffieldMarketing0 -
CSV sheets on new OSE not received by email
Is there still a problem with the new OSE anyone ? I am not receiving the csv reports I am ordering via OSE.
Moz Pro | | blocker04080 -
Best Tool to Build List of Competitor's Top Keywords?
What is the best (hopefully seomoz) tool to create a list of the keywords bringing the most traffic into a competitor's website? Goal is to build a list of keywords that bring the competitor the most traffic, stack ranked by volume of traffic coming into the competitor.
Moz Pro | | sftravel1