Moz & Xenu Link Sleuth unable to crawl a website (403 error)
-
It could be that I am missing something really obvious however we are getting the following error when we try to use the Moz tool on a client website. (I have read through a few posts on 403 errors but none that appear to be the same problem as this)
Moz Result
Title 403 : Error
Meta Description 403 Forbidden
Meta Robots_Not present/empty_
Meta Refresh_Not present/empty_
Xenu Link Sleuth Result
Broken links, ordered by link:
error code: 403 (forbidden request), linked from page(s): Thanks in advance!
-
Hey Liam,
Thanks for following up. Unfortunately, we use thousands of dynamic IPs through Amazon Web Services to run our crawler and the IP would change from crawl to crawl. We don't even have a set range for the IPs we use through AWS.
As for throttling, we don't have a set throttle. We try to space out the server hits enough to not bring down the server, but then hit the server as often as necessary in order to crawl the full site or crawl limit in a reasonable amount of time. We try to find a balance between hitting the site too hard and having extremely long crawl times. If the devs are worried about how often we hit the server, they can add a crawl delay of 10 to the robots.txt to throttle the crawler. We will respect that delay.
If the devs use Moz, as well, they would also be getting a 403 on their crawl because the server is blocking our user agent specifically. The server would give the same status code regardless of who has set up the campaign.
I'm sorry this information isn't more specific. Please let me know if you need any other assistance.
Chiaryn
-
Hi Chiaryn
The sage continues....this is the response my client got back from the developers - please could you let me have the answers to the two questions?
Apparently as part of their ‘SAF’ (?) protocols, if the IT director sees a big spike in 3<sup>rd</sup> party products trawling the site he will block them! They did say that they use moz too. What they’ve asked me to get from moz is:
- Moz IP address/range
- Level of throttling they will use
I would question that if THEY USE MOZ themselves why would they need these answers but if I go back with that I will be going around in circles - any chance of letting me know the answer(s)?
Thanks in advance.
Liam
-
Awesome - thank you.
Kind Regards
Liam
-
Hey There,
The robots.txt shouldn't really affect 403s; you would actually get a "blocked by robots.txt" error if that was the cause. Your server is basically telling us that we are not authorized to access your site. I agree with Mat that we are most likely being blocked in the htaccess file. It may be that your server is flagging our crawler and Xenu's crawler as troll crawlers or something along those lines. I ran a test on your URL using a non-existent crawler, Rogerbot with a capital R, and got a 200 status code back but when I run the test with our real crawler, rogerbot with a lowercase r, I get the 403 error (http://screencast.com/t/Sv9cozvY2f01). This tells me that the server is specifically blocking our crawler, but not all crawlers in general.
I hope this helps. Let me know if you have any other questions.
Chiaryn
Help Team Ninja -
Hi Mat
Thanks for the reply - robots.txt file is as follows:
## The following are infinitely deep trees User-agent: * Disallow: /cgi-bin Disallow: /cms/events Disallow: /cms/latest Disallow: /cms/cookieprivacy Disallow: /cms/help Disallow: /site/services/megamenu/ Disallow: /site/mobile/ I can't get access to the .htaccess file at present (we're not the developers) Anyone else any thoughts? Weirdly I can get Screaming Frog info back on the site :-/
-
403s are tricky to diagnose because they, by their very nature, don't tell you much. They're sort of the server equivalent of just shouting "NO!".
You say Moz & Xenu are receiving the 403. I assume that it loads properly from a browser.
I'd start looking at the .htaccess . Any odd deny statements in there? It could be that an IP range or user agent is blocked. Some people like to block common crawlers (Not calling Roger names there). Check the robots.txt whilst you are there, although that shouldn't return a 403 really.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New Website & DA Score
We are preparing to build a new website for a branch of our company. According to Neil Patel (whom I highly respect) our DA should come in around 30 (categorizing us as a startup, see chart below). So I ask: Why do our current sites DA Score below 30? (20 & 29 respectively) Our sites were redesigned in 2013 and are responsive, pass the Google Mobile Friendly test and I've improved load speed by over 25% by compressing images. I have used Moz as guidance for 10 months on both 7 + yo sites, HP.com is 14 yo, our pages score in the high 90's and I have disavowed bad backlinks for 6 months (now only every other month). I thought we would be in or close to the 50 range by 2017 after a solid year of improvements, but I don't see that happening. Our scores go up a point or two, then back down, we have floundered 2 points either way for 10 months. Most important question: What do I do now to raise our DA? Lastly, my plan is to take all the lessons learned from the past years and apply it to the new site, measure the success and then redesign the other two sites in accordance to the new model. But that may be a year away. KJr www.heritageprinting.com www.heritageprintingcharlotte.com | DA | Rating |
Moz Pro | | KevnJr
| 1-10 | Poor – Your site is young and weak. You have a lot of growing to do. |
| 11-20 | Decent. Your site isn’t stellar, but you’re doing better. It would be good to grow. |
| 21-30 | Fair. Your site shows signs of SEO, but there are many things you can and should do to improve. |
| 31-40 | Competitive. A lot of startups find themselves in this DA range. It’s not bad, and you’re beginning to get close to the sweet spot. |
| 41-50 | Good. Now, you’re getting somewhere. This is a nice place to be, and many good e-commerce sites find themselves squarely in this category. |
| 51-60 | Strong. As you swing out of the lower half of the scale, you’re beginning to get much healthier. This is a good place to be. |
| 61-70 | Excellent. A DA at this level represents a great site with a lot of recognition, a lot of link backs, and a considerable authority in its niche. Many .edus are in this space. |
| 71-80 | Outstanding. You’re dominating in the SERPs and owning your niche. Quick Sprout is a 73. |
| 81-90 | Very outstanding. You’re in the upper echelons of authority. You can consider yourself to have arrived. |
| 91-100 | Rare. These sites are household names — Wikipedia, Facebook, New York Times, etc. Your site will probably never attain this level. Only a miniscule fraction of a percentage of sites on the Internet ever get this high. | https://www.quicksprout.com/2014/04/09/how-to-score-your-websites-seo-in-10-minutes-or-less/0 -
I need an interlinking report for my site, is there a report in Moz or another application that tell me how all of my pages are linked to other pages on my site?
I am in the process of doing a redesign for one of my sites. I need an interlinking report for my site. Is there a report in Moz or another application that tell me how all of my pages are linked to other pages on my site?
Moz Pro | | seoflorida0 -
Website disappeared from Google :(
Dear all, I've a 5 years old website, which was in top rank for a particular keyword. It's been a couple of months that the site suddenly disappeared from Google.com.au search. However, the site is number one on Bing and Yahoo.
Moz Pro | | jarrodb
I've tried almost everything in my limited knowledge but it doesn't seem to come back. Can you please suggest if something can be done. Google.com.au
website: http://partysuppliesnow.com.au/
keyword: "party supplies" Your help would be highly appreciated.
Thanks in advance 🙂
Jarrod0 -
Crawl Diagnostics
My site was crawled last night and found 10,000 errors due to a Robot.txt change implemented last week in between Moz crawls. This is obviously very bad so we have corrected it this morning. We do not want to wait until next Monday (6 days) to see if the fix has worked. How do we force a Moz crawl now? Thanks
Moz Pro | | Studio330 -
Site Not Indexing & SEOMoz Reporting ZERO On-Page Report Crawls
Any help on this would be MUCH appreciated. One of my sites, aironeairsolutionsinc.com, has recently been rebuilt and the pages tweaked for some basic optimization. Based on my experience, those tweaks (geared toward keywords with relatively low competition locally) usually bump my local sites up into the top 20 or 30 at worst. 3 weeks later, it seems my site is still not indexing with Google. In addition, I AM NOTICING THAT THE ON PAGE REPORTS IN SEO MOZ ARE NOT REGISTERING THAT ANY PAGES ARE BEING CRAWLED. Again, any help from Moz staff would be awesome! :} Thanks, Ricky
Moz Pro | | RickyShockley0 -
Why would opensite explorer show a link from baidu result page with baidu as a linking root domain?
I found an old domain that has very high domain authority and one of its top linking root domains is baidu I clicked the link in ose and it took me to a baidu serp????? please explain...im happy to clarify if need be or give you the site in question just let me know what you think. I checked ose for the page authority of the serp page it was 50. would that page be helping the site in question to rank? or am I just dumb for asking this question One more thing I couldnt actually find a link to the page I was looking at in ose on the baidu result page.. thanks
Moz Pro | | duncan2740 -
Getting PA & DA off of a list of links
I have a list of links that I want to get PA and DA for each individual link, can this be done in some way other than one at a time? I've heard this can be done with excel and using api but I don't know the specifics of this.. Help would be appreciated
Moz Pro | | Fergclaw2