804 HTTPS (SSL) error

[empleostampico]

Hi Lucas,

You're definitively not going crazy about my name! My business partner created the Moz account and I just changed the name of moz profile to mine as I'll be managing this..hence the confusion.

I am running a wordpress install hosted with siteground and paid for cloudflare plus via siteground, all the site redirects to https as well, no mobile redirect.

I just activated IPV6 compatibility, HTTP/2 + SPDY, set firewall security level to LOW and gave the moz crawl test a try, I'll post my results tomorrow.

By the way, I just took a look at your site and I'm impressed, it's awesome, my website is a local job portal so there's a few good features you have I'm jealous about! I'm not a designer so we went with a "premium" wp template and started tweaking from there. Out of curiosity is your site a custom design solution or is there a way I can get my hands on that theme?  

Many thanks again,

Jorge

[lrafagnin]

Sorry Jorge, not sure why I called you Rodrigo...
No worries, happy to help, it drove me crazy too

So, I reviewed my CloudFlare settings, I got firewall security level set to low, no mobile redirect but I do have a redirect on HTTP to HTTPS on all pages.
I also have IPv6 on although I doubt that would make any difference.

What I can off is that your application might be blocking the requests maybe, not CF.
Do you run custom code or is it an off the shelf solution like Wordpress?
And do you run it on AWS? EC2 instances or though an ELB?

I would suggest as your next step to check if you see any requests from Moz to your server
Just trigger a Crawl test on this link and wait a few hours, you will get the report by email
https://moz.com/researchtools/crawl-test

Let us know how it goes

[empleostampico]

Hi Lucas,

Thank you for the quick response. I am using cloudflare plus at moment, and I do have a dedicated IP and SSL with my hosting provider. Would that not be the same as above? I used the dns tool and I do get a constant IP address.

Again, thank you very much for helping on this! - I understand it is not your duty to do so.

[lrafagnin]

Hi Rodrigo,

You need to be on CloudFlare Pro which is $20/month to get a dedicated IP address for SSL which will solve your problem.
You don't need to disable browser integrity but you might need to wait a few minutes until CloudFlare adds the IP to your domain. You can check it with a tool like https://www.whatsmydns.net/

[empleostampico]

Hi Lucas,

I am using cloudflare and can't get moz crawler to run even though I have disabled browser integrity and flushed cache. Is there anything else you had to do to have this work?

Many thanks

[lrafagnin]

Hi Roel,
I totally agree with you, but Moz seems to be really slow to move on this (it's been over 4 years since python released SNI support).
If you really want to use Moz I would strongly suggest just making the changes on your CDN instead of waiting for them to make a move, and you can always revert the setup once they release SNI support.
Google Cloud supports dedicated IP (they call it a VIP - virtual IP) and it costs $39/month.
You can find some info on how to setup here, I have never done it but seems pretty straight forward
https://cloud.google.com/appengine/docs/python/console/using-custom-domains-and-ssl

[crowdsite.com]

Hi Lucas,

It is possible with most hosting providers, we use Google Cloud Platform and i believe it's possible there. But it involves changing all the certificates from all sites (Our site runs on 12 domains), it's a lot unnecessary work.

Cheers, Roel

[lrafagnin]

Guys,

I don't work for Moz and I I know it sucks that they don't support SNI, even if every major browser since IE6 does it, but it isn't a hard fix to add a dedicated IP address for SSL.
If you use AWS you can get CloudFront's dedicated IP for $600/month or you can add CloudFlare Pro for $20/month.
Every hosting company I know provides dedicated IP for SSL so the only issue here is the additional cost.
You guys are paying $100/month for Moz so might as well add another $20 (per domain) and forget about it
Happy to give some guidance with the setup if anyone isn't technical, you should also be able to switch the DNS with zero downtime.

Cheers,
Lucas

[DavidLee]

The fix we are working on involves a complete re-write of our crawler which will take time to complete. I'm afraid I do not have an ETA but will keep this thread posted with any updates.

[crowdsite.com]

Hi Moz Users,

Shouldn't it be fixed by now? We got the same problem at https://www.crowdsite.com

We also use SNI on our servers, is there a ATE to fix?

[KyleMamaBird]

Is there any updates on the issues with Cloudflare?

We like Moz but really would like this resolved.

[DavidLee]

If SNI is not enabled with GoDaddy, the other issue would be related to not having intermediate SSL certificates setup.

[Pablostevens71]

I am also having this issue.  I have a GoDaddy SSL on their Managed Wordpress Hosting and just started getting the 804 error in December 2015.  Are there any other common things to check other than on GoDaddy, there's not really much to configure there and everything else looks OK.

[DavidLee]

Sounds great Lucas!

Will give these settings to another user to see if it works.

Thanks again!

[lrafagnin]

Hi David,

I did some more testing and it seems that the Crawler test was able to scan with the browser agent check on which is good news.
Moz will be indexing our site over the weekend and I can let you know then and confirm everything works as expected.

Cheers,
Lucas

[DavidLee]

Hi Lucas

Glad to hear you were able to find a workaround! One concerning bit is the disabling of the BrowserAgent security check. Would you be able to check with CloudFlare to see if this is recommended? Your findings may benefit many users that run into this issue and we can recommend this setup to other CF users.

Cheers!

[lrafagnin]

So good news on my side, I was able to get the crawler test to work last night
My current setup is

• CloudFlare on Pro plan $20/month - you need this because it sets an static IP to your website
• On CF settings, disable BrowserAgent security checks - I still need to review this further but it seems CF blocks Moz's Rogerbot crawler because they cannot verify the user agent

I'll keep you posted on any new developments I have

[SEMbyotic]

I wish I had a solution for you, but I'm still using SNI via standard CloudFlare so I'm not able to leverage the Moz crawl at all. If you find a solution, it'd be great if you can post it here for myself and the others.

[lrafagnin]

Hi there,

We upgraded our site to CloudFlare Pro which supports all browsers that have TLS 1.0 and above - it gives the domain an static IP which means no SNI.
OpenSSL also connects to the domain without any issues but no luck in getting the crawler to work.
I also disabled Browser agent check security on CloudFlare which could be causing issues but that also didn't work.

Can anyone point me in the right direction?
Is there any way to get MOZ to crawl a site hosted on CloudFlare or is this a deadend?

If someone could provide a bit of code which I can use to test and debug the crawling request without having to use the crawling test tool that would be very helpful.

Cheers,
Lucas

[bluedotproductions]

We too use SNI for our certificates on my co-located server with a top host.

It sounds like you were planning on implementing this since July 2015? It's Christmas Day - 2015, that's a long time Moz.

[ClockSoftware]

Same here.

We use CloudFront and receive the 804 error response. We consider to cancel our subscription  as we get no data and there is no information if and when a solution can be expected.

[micvitale]

same problem for me.

I am working with https and cloudflare.

what if I disable cloudflare just for the crawl test? would it work? how could I disable it temporarily?

thank you

[Arizuka]

Same problem for us. We are glad to use moz but we have migrated from http to https and nothing works. Do you have any clue if it will be or not in your roadmap developments?

[dswz]

Please fix this. I am using SNI certificates from Cloudflare on different domains. IPs are getting rare.

[CandymanKen]

Hi-

Have there been any updates to this? We have been using  cloudlfare for probably a year and all of a sudden I am getting the 804 errors on our SSL.

Any help would be appreciated.

Ken

[FabricEleven]

Is there a way around this? Maybe creating a cname in cloudflare like moz.domain.com and set it to not use their https would that work? Or will all the results be skewed since domain name has changed?

[jameskais]

At this time they will need their own as we do not support any SNI at this time. With that being said we are aware of it and have investigated solutions but an ETA is not quite yet available. If these are the only sites you planned on using Moz with we won't be of much service at this time =(.

[jameskais]

Hey there - sorry if there was a miscommunication but we do in fact support SSL, however at this time we do not support SNI (Server Name Identification) which is a technology used to host multiple SSL certificates from a single IP address. If you are currently leveraging CloudFlare or CloudFront we will not be able to crawl your site. If you are using a standard SSL from it's own IP we can most likely crawl that page.

[dsmolinski]

My understanding is that Moz can't crawl sites with SSL/Https.  I like Moz but if it doesn't work then I can't continue to pay for it then.

This was a response I received from support:

if you tried the suggestions in those posts and were unable to find a solution, then unfortunately it looks like we will not be able to crawl your page at this time. Our crawler has trouble with SSL pages and is often not able to crawl them at all. We are working towards fixing this, but it's a large-scale project. I'm sorry to be the messenger on this. If there is anything else I can help with, please let me know.

[SEMbyotic]

I too am using CloudFlare and have the same error. I'd like to be notified when you have an update.

Thanks,
Adam

[jameskais]

Hi Lane - Our current crawl libraries do not support SNI which is typically what is enabled by default by Cloud Flare. Our engineering team is still investigating expanding our libraries but an ETA is not available quite yet.

[lanehouk]

I am getting these 804 crawl errors on multiple campaigns/domains

We are using Cloudflare Flexible SSL on all these domains. Can anyone advise on this?

[donford]

Yes, you need to work with Moz support to get the issue fixed.

[Happy-SEO]

And a 804 is a MOZ error I guess? It isn't something that would show up in a normal crawler?

[Happy-SEO]

I already did, but they gave me this answer:

MOZ: (there was a question before this one)
I don't think so as the issue is not in loading it or receiving the certificate it is that we cannot translate the certificate or receive an error on getting it.Did the hosting provider state anything about SNI or serving multiple certificates from the same server?

That's a bit to technical for me.

[donford]

I just ran a crawl and did not see any 804's

You can view the results here:

http://freetools.webmasterworld.com/freeninja/google-sitemap-generator/2ba22964bda923870e58198ba6411c13.html

You may want to contact Moz directly to see if one of the Moz staff can help you further.

[Happy-SEO]

Hi Guys

I fixed the issues and the SSL is working fine, but still have an 804 error, so still can't track my campaign.

Hope you can help me further.

[donford]

Hi Jonathan,

Didn't realize this was possible now. For the reasons you stated it still would be advised to get a dedicated IP. The last time I looked at prices they were relatively in-expensive.

[TheWebMastercom]

Actually, many of the larger hosts using cPanel now do not require a dedicated IP as they use SNI technology that allows an SSL certificate to be installed on a shared IP.  However, there can be other benefits to having a dedicated IP such as being less likely to get caught on an Email blacklist etc.

[donford]

Hello Happy,

Okay so you have content being served as http on your https page. When you reference an image or script you need to make sure it is a relative reference or a https reference, otherwise you will get these types of warnings.

See Mozilla Facts here

Also see the image attached.

Also the SSL isn't misconfigured it is missing. To configure one properly you need to contact your host and ask them to install a SSL cert (most host will not allow users to do this themselves). If you have not yet purchased a SSL you will need to do so. SSL certs also require dedicated IP addresses which most host also charge for.

In summary if you purchase a dedicated IP and a SSL certification you're problem should go away unless you specifically declare content as http.

Hope this helps,

Don

Gkw1qEV.png