I got a 404 in the Crawl Test Tool Report
-
I, yesterday i ran an crawl on http://www.everlastinggarden.nl and i get an 404. Does anybody know why this happens?
<colgroup><col width="1535"></colgroup>
| # ---------------------------------------- |
| Crawl Test Tool Report | Moz,http://pro.seomoz.org/tools/crawl-test |
| www.everlastinggarden.nl |
| Report created: 15 Jul 18:34 |
| # ---------------------------------------- |
| URL,Time Crawled,Title Tag,Meta Description,HTTP Status Code,Referrer,Link Count,Content-Type Header,4XX (Client Error),5XX (Server Error),Title Missing or Empty,Duplicate Page Content,URLs with Duplicate Page Content (up to 5),Duplicate Page Title,URLs with Duplicate Title Tags (up to 5),Long URL,Overly-Dynamic URL,301 (Permanent Redirect),302 (Temporary Redirect),301/302 Target,Meta Refresh,Meta Refresh Target,Title Element Too Short,Title Element Too Long,Too Many On-Page Links,Missing Meta Description Tag,Search Engine blocked by robots.txt,Meta-robots Nofollow,Blocked by X-robots,X-Robots-Tag Header,Blocked by meta-robots,Meta Robots Tag,Rel Canonical,Rel-Canonical Target,Blocking All User Agents,Blocking Google,Blocking Yahoo,Blocking Bing,Internal Links,Linking Root Domains,External Links,Page Authority |
| http://www.everlastinggarden.nl,2014,404 : Received 404 (Not Found) error response for page.,Error attempting to request page |Best regards,
Jos
-
Erin,
thank you very much for your effort. I already contacted my host on this problem, but they say is that they are only blocking Yandex and that every other crawler is allowed.
In the past my client did have similair errors (website not reachable) but that was solved i understood. I will contact you via in private for additional information.
Also I will contact the host again and ask them to investigate the matter.
Best regards, Jos
-
Hi Jos,
Thanks for reaching out! My name is Erin, and I'm on the Moz Help Team. Our crawls have been delayed, and we're currently working through the backlog. Thanks for your patience! Keep in mind that the Crawl Test is cached for 48 hours, so if you run them too close together you'll see the same data.
I curled your site from our servers, and I came back with a 404. Usually this means that your site is blocking us on the server level- and explains why your site comes back with a 200 from a Header Checker, but a 404 from our servers. I did a little digging but wasn't able to come up with much, so you'll want to loop in your web dev to see if they can shed some more light on what might be happening. I'm sorry there's not more I can do for you on the support side with this one!
I don't want to share any private site details over the interwebz, so if you have any follow up questions email the Help Team.
I hope that helps, and happy Monday!
Cheers,
Erin
-
Hi! I've asked our help team to come and take a look at this and investigate. Thanks for your patience!
-
Can anybody help? I re-issued another crawl, but this one is already 4 days on it's way ????
See attachement.
Best regards, Jos
-
Er_Maqui, Keri,
thank you for you responses. I informed with the host, the only block Yandex everything else is allowed.
I did a re-do of the crawl with the same result ;-(
@Keri, it's a 404 on the homepage.... :
http://www.everlastinggarden.nl,2014,404 : Received 404 (Not Found) error response for page.,Error attempting to request page.
Also in my campaign it gives a 404. If i check the a-record, all looks good: https://www.whatsmydns.net/#A/www.everlastinggarden.nl
Best regsrds, Jos
-
Hi! If you download the CSV of your crawl report, it will list the referring URL that is pointing to the page with a 404.
-
Try to re-do the crawl test. This can be a puntual problem.
For getting more info, check your server log. Try to enter the page before re-do the test for verify all it's working.
Also, if you can, check if your hosting is blocking some user-agents (question it to them) for reduce traffic.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate Content & Title Tag Group Fields on MoZ Report
Hello, On my SEO MOZ exported Site Crawl CSV report, I have columns for Duplicate Content Group & for Duplicate Title Tag Group. The values in the columns are numerical - 20, 5 , 15, etc. Can anyone explain to me what these values represent and how I can fix the issues I presume they represent? Thank you,
Moz Bar | | AED-1
Scott0 -
Rogerbot will not crawl my site! Site URL is https but keep getting and error that homepage (http) can not be accessed. I set up a second campaign to alter the target url to the newer https version but still getting the same error! What can I do?
Site URL is https but keep getting and error that homepage (http://www.flogas.co.uk/) can not be accessed. I set up a second campaign to alter the target url to the newer https://www.flogas.co.uk/ version but still getting the same error! What can I do? I want to use Moz for everything rather than continuing to use a separate auditing tool!
Moz Bar | | digitalascend0 -
Crawl Report Internal Links Count
We recently ran a crawl report on www.phase1tech.com. Some of the pages are coming back with a large amount of 'internal links'. These 2 pages for example are showing 800 internal links: http://www.phase1tech.com/Upcoming-Events
Moz Bar | | AISEO
http://www.phase1tech.com/Contact At best there are approximately 70 links on the page. Where is the 800 number coming from?0 -
Moz Crawl Test Trying to Crawl Contact Form Submit Button Location?
Moz Crawl Test for some reason is trying to Crawl a contact form Widget Submit Location. My obvious guess is that obviously the crawl cannot submit to the required fields…..I believe this because they're only kicking back these errors on the pages I have a contact form widget on. http://crawfordspest.com/pest-control/crawfords@crawfordspest.com 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404
Moz Bar | | Funk-Creative-Media
http://crawfordspest.com/tree-services/crawfords@crawfordspest.com 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404
http://crawfordspest.com/lawn-care/crawfords@crawfordspest.com 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404
http://crawfordspest.com/specialty-services/crawfords@crawfordspest.com 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404 Can you shed any insight to this? I'm a bit worried that I'll have to complete gut the contact form which was one of the major requests my client requested. Or in a worse scenario make all fields not required. It would let so much spam in. I have never seem anything like this at all. But I've learned a lot from Moz, and with major errors like 404 damage Domain Authority greatly. I've fixed 404 issues with newly acquired clients existing sites and tracked through Moz and the domain authority flies up once these errors are fixed. Along with fixing what Webmaster Tools through Google reports back. ..... Let me know if you have any expertise on this matter.0 -
MOZ crawl test is not reporting on all the pages on my site.
I've run the crawl test one of the sites I've taken over SEO for, however its only picking all the pages. For instance it indexes all the pages under xxxxx/us but none under xxxxx/au or xxxxx/uk The pages are being indexed as they're ranking in Google. Thanks.
Moz Bar | | ahyde0 -
Suggestion for Improving the Crawl Report on Canonicals
This came up in the answer to a question I gave here http://moz.com/community/q/canonicals-in-crawling-reports#reply_222623 Wanted to post here to put it in as a suggestion on how to improve the Moz Crawl reports Currently, the report shows FALSE if there is no canonical link on a page and TRUE if there is. IF you get a TRUE response, this shows up as a warning in your report. I currently use Canonical to Self on almost all my pages to help with some indexing issues. I currently use the EXACT function in excel to create a formula to see if my canonical link matches the URL of the page (as this is what I want it to do). I can then know that the canonical is implemented properly, or if I need to manually check pages to make sure the canonical that points to another page is correct. I would like to suggest that the Moz crawl tool does this. It can show FALSE is the canonical is missing, TRUE if the canonical is present and SELF if the canonical points to the URL of the page it is on. I think for the most part this would be much more actionable information. I would even suggest that TRUE would need to be more of a high priority alert, and SELF can't do any damage, so I would leave that info in the CSV but not have that as a warning in the web interface. Thanks for listening!
Moz Bar | | CleverPhD0 -
Dupe content report showing in 'Errors' section when surely should be in 'Warnings' section ?
Why is the dupe content info showing in errors and not warnings ? Since if dupe content can get your site penalised (as per Panda) or worse banned, surely it should be in that section of reports ? Cheers
Moz Bar | | Dan-Lawrence
Dan0