Welcome to the Q&A Forum

The crawl test currently fails, it just says error submitting the crawl test. Very odd.

Hi Toby,

All sites are SSL only (or with a redirect that makes it so). Seems that all our CF Pro sites are now being crawled.

Lee

Interestingly, I do have some of our sites on CF Pro ($5/month plan) that are being crawled, but some not (yet at least). Will post if this changes.

This was the response I got:

From Moz support:

Apologies about this! Unfortunately, the version of python we use does not support SNI and in order for us to upgrade to a version that does, it involves completely re-writing our crawler which can take some time. We have learned of more sites running into this issues, so it is a priority. I can't say exactly when we will have an update. Sorry for the inconvenience!

---

Not ideal, so I queried them about us having to pay $/month to Cloudflare to overcome a technical issue that should be supported (IMO) by Moz.

From Me:

Thanks for this: now I understand why it's not done.  the only things I have to say in reply are:

1. We now have to pay $XXX/month to Cloudflare mode that we need to in order for Moz to crawl the sites. Can you please arrange to compensate us for this in some fashion? Either some more projects available to us, or perhaps a couple of months free? I really don't believe it's fair to ask us to pay more on other services in order for Moz to be able to spider our website, considering what we are doing is totally standard.

2. The non-SLI issue has been going on for months, if not years. So cannot really be a priority for Moz. I'm not the only one stunned on the forums.

---

From Moz support:

I'm afraid we don't offer compensation for these SNI issues, unfortunately. It's a limitation of the technology right now. Rewriting crawlers takes a significant amount of time and effort, and although it is high on our priority list I still don't have any estimated release dates of when you might see a new one. Sorry about this!

Ditto. We've had to upgrade all our CF sites to pro, which costs us. SNI is so widely supported in the fabric of the Internet that it's really quite foundational to support it.

It's been a year now guys. What's the score with supporting it?

This was the response I got:

From Moz support:

Apologies about this! Unfortunately, the version of python we use does not support SNI and in order for us to upgrade to a version that does, it involves completely re-writing our crawler which can take some time. We have learned of more sites running into this issues, so it is a priority. I can't say exactly when we will have an update. Sorry for the inconvenience!

---

Not ideal, so I queried them about us having to pay $/month to Cloudflare to overcome a technical issue that should be supported (IMO) by Moz.

From Me:

Thanks for this: now I understand why it's not done.  the only things I have to say in reply are:

1. We now have to pay $XXX/month to Cloudflare mode that we need to in order for Moz to crawl the sites. Can you please arrange to compensate us for this in some fashion? Either some more projects available to us, or perhaps a couple of months free? I really don't believe it's fair to ask us to pay more on other services in order for Moz to be able to spider our website, considering what we are doing is totally standard.

2. The non-SLI issue has been going on for months, if not years. So cannot really be a priority for Moz. I'm not the only one stunned on the forums.

---

From Moz support:

I'm afraid we don't offer compensation for these SNI issues, unfortunately. It's a limitation of the technology right now. Rewriting crawlers takes a significant amount of time and effort, and although it is high on our priority list I still don't have any estimated release dates of when you might see a new one. Sorry about this!

The crawl test currently fails, it just says error submitting the crawl test. Very odd.

Interestingly, I do have some of our sites on CF Pro ($5/month plan) that are being crawled, but some not (yet at least). Will post if this changes.