Different Errors Running 2 Crawls on Effectively the Same Setup

[I-COM]

Our developers are moving away from utilising robots.txt files due to security risks, so e have been in the process of removing them from sites. However we, and our clients still want to run Moz crawl reports as they can highlight useful information.

The two sites in question sit on the same server with the same settings (in fact running on the same Magento install). We do not have a robots.txt files present (they 404), and as per Chiaryn's response here https://moz.com/community/q/without-robots-txt-no-crawling this should work fine?

However for www.iconiclights.co.uk we got: 902 : Network errors prevented crawler from contacting server for page.

While for www.valuelights.co.uk we got: 612 : Page banned by error response for robots.txt.

These crawls were both run recently, and there was no robots.txt present. Not to mention, they are on the same setup/server etc as mentioned. Now, we have just tested this, by uploading a blank robots.txt file to see if it changed anything - but we get exactly the same errors.

I have had a look, but can't find anything that really matches this on here - help would really be appreciated!

Thanks!

[tawnycase]

Hey there! Tawny from the Customer Support team here!

This sounds like a juicy issue, and one I'd love to dive in and help you with! Unfortunately, without being able to take a look at your campaigns and account directly, it's tough to provide specific support for these issues.

That said, if you write in to help@moz.com and give us the details of what you're seeing - basically exactly what's in this question - we should be able to help investigate for you.

[MikeRoberts]

Having no Robots.txt, or a blank one, is perfectly fine (though honestly its no more a security risk than your Sitemap.xml). But your current issue is that both of your sites are returning 403 status codes at crawlers while people are still able to land on your pages. This has nothing to do with the Robots.txt file being changed or removed; just an odd coincidence. This most likely is an issue in htaccess file.