Rogerbot will not crawl my site! Site URL is https but keep getting and error that homepage (http) can not be accessed. I set up a second campaign to alter the target url to the newer https version but still getting the same error! What can I do?
-
Site URL is https but keep getting and error that homepage (http://www.flogas.co.uk/) can not be accessed. I set up a second campaign to alter the target url to the newer https://www.flogas.co.uk/ version but still getting the same error! What can I do? I want to use Moz for everything rather than continuing to use a separate auditing tool!
-
It will probably help if you delete the following line from your robots.txt file:
User-Agent: rogerbot
Disallow: /Dirk
PS You also block /js/ for all bots - Google doesn't really recommend that
-
I just check and see that http is 301 to https. But https works.
Can you mail this issue to help@moz.com?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
On Page Grader - URL not accessible
We have tried to use the On Page Grader today and it is coming back with URL not accessible for all pages on our website. We previously used the On Page Grader on Friday 10th Nov for a couple of product pages with no issues. Since then, the only changes we have made on the websites is updating some downloadable documents. We have done this several times before and it has never affected Moz. We have not changed the page URLs, and therefore do not know why it is now not working. The pages are working fine on the website with no issues. A link to one of the pages is below. http://www.processinstruments.co.uk/products/dissolved-oxygen-monitor/ Any help would be greatly appreciated.
Moz Bar | | PiMike0 -
I have recently switzed to SSL on ramxpert.dk, now Moz is not able to crawl the site anymore
Hi Moz I have recently switzed to SSL on ramxpert.dk, now Moz is not able to crawl anymore. When I look at the crawl report there is just one fault displayed: "403 : Received 403 (Forbidden) error response for page." How can i solve this issue asap? I was not aware that there would be any issues with moz when activating a SSL certificate on the site. The campaign with the problem is: ramxpert.dk. Regards
Moz Bar | | WebBoost
Jesper Nielsen0 -
My page cant be crawled by MOZ
Hi everyone, I have a page named zita.vn. Im using Moz to do SEO for the page, but when I use On-page grader in MOZ, I got an error that the page cant be accessed. Additionally, I got an issue with page crawling. Google search still crawl my page normally. Please help. Thanks a lot.
Moz Bar | | zita.vn0 -
Onpage grader says site unreachable
It is possible the crawler is caught in the firewall. Any idea what I need to have whitelisted?
Moz Bar | | TheWebMastercom0 -
How can I add another email address so they can receive Fresh Explorer alerts?
How can I add another email address so they can receive Fresh Explorer alerts?
Moz Bar | | intelligencebank0 -
Is your MOZbar still working?
Mine does not. Only the social media signals still report. Reinstalled, restarted, all the basic troubleshooting Any ideas?
Moz Bar | | Grobbie0 -
I am doing the On-page SEO for a website that's never had any SEO done before. I will start with the Pages. Is it necessary to do SEO/Keywords for older Posts?
Using the On-Page Grader to perform SEO on pages of WordPress website. This website never had any SEO done before. Should I go back and perform the same work on the Post Pages?
Moz Bar | | Joseph.Lusso0 -
Is Manual Crawl Test option available now to Pro Users?
Hi all, I have worked on my Crawl Issues and want to see how many still exist. Earlier I was using Manual Crawl Test. However, now I don't see this tool in Moz Account. Please suggest. Thanks
Moz Bar | | chandman0