Error 406 with crawler test
-
hi to all. I have a big problem with the crawler of moz on this website: www.edilflagiello.it.
On july with the old version i have no problem and the crawler give me a csv report with all the url but after we changed the new magento theme and restyled the old version, each time i use the crawler, i receive a csv file with this error:
"error 406"
Can you help me to understan wich is the problem? I already have disabled .htacces and robots.txt but nothing.
Website is working well, i have used also screaming frog as well.
-
thank you very much Dirk. this sunday i try to fix all the error and next i will try again. Thanks for your assistance.
-
I noticed that you have a Vary: User-Agent in the header - so I tried visiting your site with js disabled & switched the user agent to Rogerbot. Result: the site did not load (turned endlessly) and checking the console showed quite a number of elements that generated 404's. In the end - there was a timeout.
Try screaming frog - set user agent to Custom and change the values to
Name:Rogerbot
Agent: Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)
It will be unable to crawl your site. Check your server configuration - there are issues in how you deal with the Mozbot useragent.
Check the attached images.
Dirk
-
nothing. After i fix all the 40x error the crawler is always empty. Any other ideas?
-
thanks, i'm wait another day
-
I know the Crawl Test reports are cached for about 48 hours so there is a chance that the CSV will look identical to the previous one for that reason.
With that in mind, I'd recommend waiting another day or two before requesting a new Crawl Test or just waiting until your next weekly campaign update, if that is sooner
-
i have fixed all error but csv is always empty and says:
http://www.edilflagiello.it,2015-10-21T13:52:42Z,406 : Received 406 (Not Acceptable) error response for page.,Error attempting to request page
here the printscreen: http://www.webpagetest.org/result/151020_QW_JMP/1/details/
Any ideas? Thanks for your help.
-
thanks a lot guy! I'm going to check this errors before next crawling.
-
Great answer Dirk! Thanks for helping out!
Something else I noticed is that the site is coming back with quite a few errors when I ran it through a 3rd party tool, W3C Markup Validation Service and it also was checking the page as XHTML 1.0 Strict which looks to be common in other cases of 406 I've seen.
-
If you check your page with external tools you'll see that the general status of the page is 200- however there are different elements which generate a 4xx error (your logo generates a 408 error - same for the shopping cart) - for more details you could check this http://www.webpagetest.org/result/151019_29_14E6/1/details/.
Remember that Moz bot is quite sensitive for errors -while browsers, Googlebot & Screaming Frog will accept errors on page, Moz bot stops in case of doubt.
You might want to check the 4xx errors & correct them - normally Moz bot should be able to crawl your site once these errors are corrected. More info on 406 errors can be found here. If you have access to your log files you could check in detail which elements are causing the problems when Mozbot is visiting your site.
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
804 : HTTPS (SSL) error encountered when requesting page.
Hi there, I am attempting a crawl test and am getting "804 : HTTPS (SSL) error encountered when requesting page." with no further information to debug the issue. Site seems to be fine as far as SSL configuration goes. How can we get more debugging information? Thanks,
Moz Bar | | RickEH
Rick0 -
Moz Crawl Report Increase in Errors?
Has anyone else noticed a huge increase over the past couple weeks in crawl issues in their dashboards? Without being able to see historical data week over week, I can't tell what's been added. Is this some update with the tool? I'm not seeing any health issues with this feature on the Moz Health page, it just seems strange that I'm seeing this across all our accounts.
Moz Bar | | WWWSEO0 -
Find SEO errors
Hi, I have a Moz Pro account. Is there any way to automatically find images without ALT tag, and also noindex/nofollow pages? Cheers,
Moz Bar | | viatrading10 -
I'm getting an error when I try to preview my custom report
I'm getting the following error when I attempt to preview my custom report. I have also attached a screenshot. The moz.com page isn’t working moz.com sent an invalid response. ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITIONI need to send my client his report for a meeting tomorrow morning, and I am unable to run it. The campaign is propacusa.comf3VhmfU
Moz Bar | | chill9860 -
Crawl Test Takes Long Time
Hi Moz, I have submitted our website for a crawl test. Usually it would only take a few hours to do the crawl. However this time, it takes quite long time and the result still shows in progress 😞 This is a small website which only contains less than 10 pages. Just wondering if this is our website setting issue or it is a technical issue at your end? Many thanks in advance. sFjAERG.png
Moz Bar | | russellbrown0 -
How can I find the old ERRORS and WARNINGS report in the NEW Moz design?
I'm looking for a complete list of errors and bugs that need to be fixed within a website. I used to use the MAIN tool (at least it seemed it was the most popular) but now that its just MOZ.com I can't seem to find that great report. It had data such as: 1. List of pages with Title Tags too long 2. List of pages with Description Tags too long 3. List of RED errors and YELLOW warnings, BLUE somethings... etc... Ring a bell? I LOVED this report, where can I find this data? Thanks! Derek
Moz Bar | | DerekM42420 -
Is Manual Crawl Test option available now to Pro Users?
Hi all, I have worked on my Crawl Issues and want to see how many still exist. Earlier I was using Manual Crawl Test. However, now I don't see this tool in Moz Account. Please suggest. Thanks
Moz Bar | | chandman0 -
Dupe content report showing in 'Errors' section when surely should be in 'Warnings' section ?
Why is the dupe content info showing in errors and not warnings ? Since if dupe content can get your site penalised (as per Panda) or worse banned, surely it should be in that section of reports ? Cheers
Moz Bar | | Dan-Lawrence
Dan0