Does Rogerbot support SNI? - And if not All sites behind Cloudflare are probably uncrawlable.

[N1ghteyes]

Hi All,

Recently i've noticed that Moz can no longer crawl any of my wbsites that are behind cloudflare. After speaking to CF support, we have come to the conclusion that a recent change by CF to force SNI for all SSL domains is the issue.

As this is likely to affect almost everyone using Cloudflare could anyone please confirm this?

In the mean time, one of the websites currently blocked is on CF enterprise, so we've requested SNI be turned off. Will test crawl again when thats done to confirm SNI is the issue.

[JaredBroussard]

We are having issues with crawls for all HubSpot hosted sites as well. We love Moz so we are hoping there is a workaround for the issue. If anyone has an update on this topic, I am all ears. Thank you.

[acreedon]

Hi James, this is also a big problem for us. Can you add me to the beta group - aine@npqmag.org? Otherwise we may have to go elsewhere for our SEO tools.

[Rachelle_Henderson]

Hi James, thank you so much! We are seeing issues with rankings for our 2 accounts using Cloudflare. I gave myself a heart attack thinking that the client's site tanked and then dug around to find the issue in the forums. It may be worth it to create a blog to let the public know what's going on so other webmasters don't freak out too since it seems a lot of us are running into this error.

[jameskais]

Hi Kent, I've added your name to our beta crawl list and hopefully we can get you added shortly. We are well underway with that re-write and will get you on a crawl that works for you shortly!

[jameskais]

Hi Rachelle, I've added your name to our beta crawl list and hopefully we can get you added shortly. As a note SNI does not impact ranking collections but only site crawls. Please let help@moz.com know if you are seeing issues with rankings collections.

[Rachelle_Henderson]

Hi! Has this been taken care of yet? It's February 2017 and we also use cloudflare, and right now I can't pull any rankings since Moz can't crawl our sites. We love Moz and do not want to cancel the service, but this is ridiculous.

[kent.dagnall]

+1 We must also use another service if Moz cannot fix their crawler issues. Roger should be one of the best bots out there for what Moz charges. Come to find out it's built on an ancient Python version and needs to be completely rewritten.

[Pentangle]

Raised in "Jun 03, 2015" and still it doesn't work?

Seriously need sorting or we'll be off to a competitor.

[orlando-rehash]

Any updates to this? I just started the Moz 30-day trial, and if this isn't available (I'm not going to pay for Cloudflare pro), I'm going to cancel Moz when my free 30 days are up.

[a_w00kie]

Is there an alternative to Moz that supports SNI ? not being able to crawl SNI enabled websites is a major problem.

[Senson]

Just want to point out that all modern browsers support SNI and more and more sites will be using it as Google and Mozilla start to make their browsers warn users of unencrypted sites.

That this issue hasn't been fixed since it was brought to Moz.com's attention is disappointing.

[MaxxDesign]

Hi, OP here (on a different Work account!)

If they upgrade to the pro level, CF SSL certs should now auto fallback for non SNI requests.

Not ideal, and Moz still need to fix the underlaying issue with the out of date version of Python that Rogerbot is written in... but it should provide a fix.

[Virtualsolutions]

Is there an update or ETA for this fix? I have a client hosted on GoDaddy and cannot crawl the site due to CF

[SYG]

Been a long time MOZ user and just found out this was an issue after implementing SSL on Hubspot.com

They are hosting our site and offered a no cost option to activate SSL, which I did, and now MOZ cannot crawl any longer.

Now anyone who is hosting their site at Hubspot and activates SSL will not be able to use Moz to crawl their site

Can't believe this is not being fixed asap by MOZ

[Quotall]

Hi All,

I've just found out that all GoDaddy Wordpress Managed websites (since Q3 2015) now have SNI enabled by default and it cannot be switched off.  Many of our clients use this hosting.

C'mon Moz - the world has moved on and we need this fixing fast.

[MaxxDesign]

Hmm ok, thats something to go on then. Moz Support couldn't crawl when i asked them to check, and gave me nothing conclusive to work with other than 'check your logs' which were completely empty (no hits in access log or against CF firewall logs)

Guessing the crawltest is just broken at the moment then...

I'll setup a campaign and see if it works. Will update when i know one way or another.

Thanks!

[leetracey]

The crawl test currently fails, it just says error submitting the crawl test. Very odd.

[MaxxDesign]

Hi Lee,

Thats interesting.. Can you use the Crawl test tool on them? or does that fail?

Currently i have no campaignes setup as i just wanted to confirm if i could crawl or not. The crawl test tool fails for me with both SSL and non SSL sites on CF pro accounts...

[leetracey]

Hi Toby,

All sites are SSL only (or with a redirect that makes it so). Seems that all our CF Pro sites are now being crawled.

Lee

[MaxxDesign]

Hi Lee,

OP here (Work account rather than personal),

Are the ones not being crawled behind SSL? As far as we can make out, Cloudflare should fall over to support non SNI crawlers for all CF Pro and above accounts. It seem however that RogerBot is spoofing a modern UA but not actually supporting modern SSL. The end result is that CF doesnt fall back properly and Moz can't crawl.

As i post this, im waiting on a Moz support response to a non SSL site behind CF thats not being crawled. Will post up if we work out what that is.

[leetracey]

Interestingly, I do have some of our sites on CF Pro ($5/month plan) that are being crawled, but some not (yet at least). Will post if this changes.

[leetracey]

This was the response I got:

From Moz support:

Apologies about this! Unfortunately, the version of python we use does not support SNI and in order for us to upgrade to a version that does, it involves completely re-writing our crawler which can take some time. We have learned of more sites running into this issues, so it is a priority. I can't say exactly when we will have an update. Sorry for the inconvenience!

---

Not ideal, so I queried them about us having to pay $/month to Cloudflare to overcome a technical issue that should be supported (IMO) by Moz.

From Me:

Thanks for this: now I understand why it's not done.  the only things I have to say in reply are:

1. We now have to pay $XXX/month to Cloudflare mode that we need to in order for Moz to crawl the sites. Can you please arrange to compensate us for this in some fashion? Either some more projects available to us, or perhaps a couple of months free? I really don't believe it's fair to ask us to pay more on other services in order for Moz to be able to spider our website, considering what we are doing is totally standard.

2. The non-SLI issue has been going on for months, if not years. So cannot really be a priority for Moz. I'm not the only one stunned on the forums.

---

From Moz support:

I'm afraid we don't offer compensation for these SNI issues, unfortunately. It's a limitation of the technology right now. Rewriting crawlers takes a significant amount of time and effort, and although it is high on our priority list I still don't have any estimated release dates of when you might see a new one. Sorry about this!

[Aurelien_TRF]

Hey guys,

Any news on this feature? or at least an ETA

[leetracey]

Ditto. We've had to upgrade all our CF sites to pro, which costs us. SNI is so widely supported in the fabric of the Internet that it's really quite foundational to support it.

It's been a year now guys. What's the score with supporting it?

[dynport]

in response to Toby New, Bing Webmastertools/Crawler seems to handle the SNI well for us.

[dynport]

Same here with AWS/Cloudfront. Please make an update. SNI is wildly supported and seems to have no impact on Google Bot or similar.

[vascoas]

We are also running in the same problem. @James Kaiser, Could you please followup the status of this issue? Do you have any release date for this fix? tks.

[MrBlastit]

Is there an ETA for SNI support as yet?

[N1ghteyes]

Hi James,

Thanks, I know it's not a 1 day fix, it's just somewhat frustrating to find that Moz doesn't yet support technology that is fast becoming standard, and thats been around for years. I look forwards to hearing when Moz does have an ETA for this! The Crawler is the main reason I created a Moz account, so i can't emphasize enough how much of an effect not having a long term solution at least announced soon is going to have on my decision to remain a Moz subscriber in the long term.

If it makes any difference, turns out Bing doesn't support it either, so something else to remember as more people start seeing crawler errors / lack of rankings that they may not understand!

http://webmasters.stackexchange.com/questions/77267/bing-and-lack-of-sni-support

And thank you for confirming the issue :).

[jameskais]

Toby - while I am unable to choose the prioritization we have actively investigated this and it is now with our product team for a resolution. This is correct that it is basically affecting all cloudflare users so it is large pain point so it will be prioritized as such.

I do wish we could get this fixed in a day but it is a larger project that needs to be addressed as such.

[N1ghteyes]

The main problem here is that i can't disable SNI unless i pay for a business account ($200 USD per website, per month), except for the enterprise account everything else is Pro which is more than enough for my clients (no way i could justify $200 a month just so moz can crawl).

SNI is a technology thats been supported since IE7 and is now standard in all browsers, so still not supporting it now is just a little insane....

But as i said, this is going to be effecting everyone using SSL under cloudflare (which is probably everyone using cloudflare) so could you guys please make this a priority? its not just an edge case issue.

[jameskais]

Hello there! At this time we are not able to support SNI but it is on the table. We have to give Rogerbot a few tweaks and some internal upgrades before he is ready to crawl these pages. In the meantime you can disable SNI or forego crawls until we've got this rolled out. No ETA has been decided for this yet.