I have recently switzed to SSL on ramxpert.dk, now Moz is not able to crawl the site anymore
-
Hi Moz
I have recently switzed to SSL on ramxpert.dk, now Moz is not able to crawl anymore.
When I look at the crawl report there is just one fault displayed:
"403 : Received 403 (Forbidden) error response for page."
How can i solve this issue asap?
I was not aware that there would be any issues with moz when activating a SSL certificate on the site.
The campaign with the problem is: ramxpert.dk.
Regards
Jesper Nielsen -
Hi Jesper,
Jo here from the Moz support team.
As WebBoost has noted our crawler isn't compatible with SNI, but the good news is that our team has been working hard on a new crawler which is in Beta right now and can handle SNI. If you're interested in being a part of the beta testing, the link to the signup page is here: http://goo.gl/forms/LCvL9Ix8JDHfbAvr1. Once you fill out that form, you will automatically be added into the Beta during the next round.
We will be rolling the new crawler out to all of our users once it's out of it's Beta phase, but I don't have an exact timeframe of when that will be at this time as it's still being worked on.
You're welcome to reach out to us at help@moz.com if you get stuck :]
Cheers!
Jo -
Thanks BeadedSEO
It seems through that the issue is that ramxpert.dk uses SNI, which isn't supported by the moz crawler yet
I've decided to sign up for the beta version of the new crawler, which I found on the link below. Hopefully this will work while they work on the final version.
https://moz.com/community/q/moz-pro-our-web-crawler-and-sites-that-use-sni
-
Have you tried updating your robots.txt file to use this;
User-agent: rogerbot
Disallow:
User-agent: dotbot
Disallow:
It's best practise to Disalow: none, rather than Allow: /
I've tried crawling your site with SEO Spider and ran in to no issues.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
In Moz Campaigns, how are competitor domains tracked if they redirect their site?
Hello! One of our competitors (Company A) that we've tracked in Moz for a long time recently merged with another company (Company B) and redirected their whole site to Company B's site. Will our competitor tracking still work as-is? Or do we need to make an adjustment? I'm reluctant to delete Company A from our competitor tracking, because we will lose all of that data. But if all of the keywords are slowly going to drop off as Google starts showing Company B results only, it may be the only option. Any help is appreciated! Thanks!
Moz Bar | | PrimeFoodTeam0 -
Can Moz Keyword Explorer help target keywords for Google Images results?
I'm wondering if I can use Keyword Explorer (or maybe another tool?) to target keywords for image rankings. I'd like to play around with optimizing images so that they appear in search results and thus provide traffic - but wasn't sure the best way to track that kind of progress. My ultimate goal is to analyze the difficulty of ranking for a certain keyword via Google images. (I do know to optimize alt tag/title tag/place in relevant article etc, but wanted to know if I could research the difficulty). Any help is much appreciated. Thanks!
Moz Bar | | naturalsociety0 -
Moz Crawl Report Increase in Errors?
Has anyone else noticed a huge increase over the past couple weeks in crawl issues in their dashboards? Without being able to see historical data week over week, I can't tell what's been added. Is this some update with the tool? I'm not seeing any health issues with this feature on the Moz Health page, it just seems strange that I'm seeing this across all our accounts.
Moz Bar | | WWWSEO0 -
I update content and then craw but the MOZ spider still shows old content. Do I need to update something else?
"This shows but was replaced a day before I ran Moz crawer: | We provide a full service for low cost automated phone calls, robocalls, Bulk SMS service, Political robo calls without needing computer skills | "
Moz Bar | | ThomasDaBomb
I look in the link on website and see:
<title>Our customers talk about: Currently the tremendous growth of organi</title> Why does the craw not reflect the current content? Thanks.
Thomas0 -
Screaming Frog, Moz and other crawlers
Hi Ignorant question, but is it possible to use Screaming Frog or the Moz crawler or any other reputable crawler for a site still in development i.e. it is yet to be indexed? If so, could someone provide some quick instructions on how this can be done. Thanks in advance for any support. Neil
Moz Bar | | mccormackmorrison0