What's the best way to eliminate "429 : Received HTTP status 429" errors?
-
My company website is built on WordPress. It receives very few crawl errors, but it do regularly receive a few (typically 1-2 per crawl) "429 : Received HTTP status 429" errors through Moz.
Based on my research, my understand is that my server is essentially telling Moz to cool it with the requests. That means it could be doing the same for search engines' bots and even visitors, right? This creates two questions for me, which I would greatly appreciate your help with:
-
Are "429 : Received HTTP status 429" errors harmful for my SEO? I imagine the answer is "yes" because Moz flags them as high priority issues in my crawl report.
-
What can I do to eliminate "429 : Received HTTP status 429" errors?
Any insight you can offer is greatly appreciated!
Thanks,
Ryan -
-
I have a customer that is using GoDaddy website hosting (at least according to BuiltWith) and I'm experiencing this same issue.
Any updates on this experiment from user rsigg? I'd love to know if I can remove this from my customer's robots file...
FWIW, Netrepid is a hosting provider for colocation, infrastructure and applications (website hosting being considered an application) and we would never force a crawl delay on a Wordpress install!
Not hating on the other hosting service providers... #justsayin
-
I am also on the same hosting and they have not been able to help with the 429. I have now started getting 429 errors when I attempt to login. Definitely something wrong with wp premium hosting.
-
Interesting. I look forward to hearing your results, as my robots.txt file is also set to:
Crawl-delay: 1.
-
We host on Media Temple's Premium WordPress hosting (which I do not recommend, but that's another post for another place), and the techs there told me that it could be an issue with the robots.txt file:
"The issue may be with the settings in the robots.txt file. It looks fine to me but the "Crawl-delay" line might be causing issues. I understand. For the most part, crawlers tend to use robots.txt to determine how to crawl your site, so you may want to see if Moz requires some special settings in there to work correctly."
Ours is set to:
Crawl-delay: 1
I haven't tried changing these values yet in our file, but may experiment with this very soon. If I get results, I'll post back here as well as start a new forum thread.
-
Chase,
They ran a bunch of internal diagnostic tools on my site, and were unable to replicate the 429 errors. They ended up telling me exactly what they told you. I haven't noticed any issues with my site's rankings, or any flags in Webmaster Tools, so it looks like they are right so far. I just hate logging into Moz and seeing all those crawl errors!
-
What'd they say Ryan? Having the same issue and just contacted Godaddy who told me that basically Moz's software is pinging my client's server too frequently so Godaddy is temporarily blocking their IP. They said it's not a concern though as they would never block Google from pinging/indexing the site.
-
Many thanks - I will contact them now!
-
Contact your host and ask let them know about the errors. More than likely they have mod_sec enabled to limit request rates. Ask them to up the limit that you are getting 429 errors from crawlers and you do not want them.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Which kind of backlink is best for SEO
After complete the jobs of keywords, it may worth to spend more time for building up more backlinks to enhance the SEO, so do you have any suggestion about which kind of backlink is best to build? as I know now, the backlinks are classified to be: 1. profile article, 2. book mark, 3. signature, 4. wiki article, 5. blog comment etc. I am very new for SEO, so it's welcome to give me any suggestion or comment for my opinion as above. Thanks a lot! -John.
Moz Pro | | Steplead0 -
Branded Keywords: Should I Monitor My Products' Names?
I am setting up my keywords' tags and brand rules. Is there any reason not to consider my products' names as "branded keywords". For instance, if my brand is HealthNut and I have products (that people talk about by name), such as GreenGiant Juice, should I create a brand rule for "GreenGiant Juice"? Granted, most of the time, customers will be searching for "HealthNut GreenGiant Juice", but there must be at least a minority who would search directly for the product itself without using the brand name. Any thoughts are greatly appreciated. Thank you in advance!
Moz Pro | | EliteErikSEO0 -
Best SEO Tool Set ! What Are they ?
Hey Moz Family. I wanted to get everyones opinion on some Seo Tools Sets that they have used and have had success with or find essential to Seo Campaigns. I have been playing around with a lot of tool sets. Just need some honest input on some tools you guys use, or a combination of tools. Best of luck, Hampig M Ceo BizDetox
Moz Pro | | BizDetox0 -
How do you guys/gals define a 'row?'
I have a question about calls to the API and how these are measured. I noticed that the URL Metrics calls allow a batch of multiple URLs. We're in a position where we need link data for multiple websites; can we request a single row of data with link information for multiple URLs, or do we need to request a unique row for each URL?
Moz Pro | | ssimburg0 -
4xx status code a page that cannot be accessed..
All the error is because the Danish letters "åøæ" in the url.. But i can access all pages, therefor the error isent a true error!!! Why is it an Issue, Google can read "øæå", why cant SEOMOZ??
Moz Pro | | seopeter290 -
Where do these URL's come from?! (Indexation issues)
We have an international webshop with languages in the URLs. Our URLs are now set up as follows: http://thermalunderwear.eu/eng/category/product Now, we know that there's some kind of strange redirect problem causing problems with our indexation, this is a technical issue that should be fixed soon. But whether this is the cause of some other strange problems, I do not know. I'd be happy with any help/advice/tips. 1. The SEOmoz site crawler starts at http://thermalunderwear.eu. This currently does not yet redirect to http://thermalunderwear.eu/eng like we want it to, but all the links on the page do include the default language code. So all links on the page are http://thermalunderwear.eu/eng/category etc. However, apart from those URLs, the site crawler finds many URLs in the form http://thermalunderwear.eu/category/product etc., so not including the language variable. Where it gets these I do not know, and since these URLs dont exist and the webshop simply shows the homepage, these URLs all have 50+ duplicate titles/content. Why oh why? 2. If I do a Google search for indexed URL's with English as language, I get many results formatted like this: Coldpruf Enthusiast mens thermal shirt - Thermal wear for men ...
Moz Pro | | DocdataCommerce
thermalunderwear.eu/eng/men/coldpruf-enthusiast-mens-thermal-shirt 170+ items – Fine-ribbed longsleeve thermal shirt men from Enthusiast ... {$SCRIPT_NAME} eng/men/coldpruf-enthusiast-mens-the {$ajax_url} http://thermalunderwear.eu/ajax What are those variables doing there? It looks like it's taking something from our Smarty debug console, which is hidden but still active in the source code, but also the ajax URL which is in a completely different location. What is Google trying to show here?0 -
How to tell where a competitor's Facebook Shares are coming from?
Is there a way to tell where a competitor's facebook shares and likes are coming from? How to also tell what ads they have running in Facebook and their spend? I am looking at OSE's data and its telling me a competitor has 8.5K Facebook shares and 1K facebook likes. I go to their facebook page and it has about 25 likes. This site should not have anywhere close to the facebook shares/likes its receiving so wondering where they are getting their boosted traffic from.
Moz Pro | | rjb6270 -
How long should "Term Extractor" be let alone to complete??
It's been left alone to grind now for a good 20 mins, and ah... it's still grinding away. Is it worth just trying again? Or does that typically mean it probably isn't going to work on the domain it's, ah, grinding away on? Thanks
Moz Pro | | iansears0