Welcome to the Q&A Forum

Hi Lee,

OP here (Work account rather than personal),

Are the ones not being crawled behind SSL? As far as we can make out, Cloudflare should fall over to support non SNI crawlers for all CF Pro and above accounts. It seem however that RogerBot is spoofing a modern UA but not actually supporting modern SSL. The end result is that CF doesnt fall back properly and Moz can't crawl.

As i post this, im waiting on a Moz support response to a non SSL site behind CF thats not being crawled. Will post up if we work out what that is.

Hmm ok, thats something to go on then. Moz Support couldn't crawl when i asked them to check, and gave me nothing conclusive to work with other than 'check your logs' which were completely empty (no hits in access log or against CF firewall logs)

Guessing the crawltest is just broken at the moment then...

I'll setup a campaign and see if it works. Will update when i know one way or another.

Thanks!