We recently switched from HTTP to HTTPS and we are having crawling issues!
-
We switched our website from HTTP to HTTPS and we started to get an email from Moz about the robots.txt being unable to crawl our website. The website is hosted through wordpress but we haven't had any issues until we switched. We have no idea what to do or even what the problem is! If you have had a similar problem and fixed it, we need your help! Thank you.
-
I know this is an old thread, but we are still having the same problem. I finally got around to sending a note to flywheel about this problem and it came back that everything is fine. I am not sure what to do here? It's on a shared hosted, so I don't have console\audit log access, however Flywheel is one of the best wordpress hosting companies out there (only thing they do).
As far as accessing the robots.txt file, I can go directly to it without any problems?
https://southernil.com/robots.txt -
Hi there!
Thanks so much for reaching out! I'm sorry you're having trouble!
I took a look at your crawl data and your site to see if I could figure out the issue. When I first tried to access your robots.txt file from a browser, it returned an error saying there were too many redirects in place. I checked to see what our crawler was receiving from your server and looks like it keeps being served a 301 redirect which points back to itself. However, when I tried to access the file from a browser a bit later, it loaded without a problem. I'm wondering if you can check your server logs to see what your server is sending back to our crawler Rogerbot?
If you could send any further info over to help@moz.com, that would be great! That way we can do some more digging and see what's going on.
Looking forward to hearing from you!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz site crawl doesn't work
The Moz site crawl isn't working for my campaign, but works for the site's on demand crawl. The search should not be disallowed by robots.txt or the headers. I'd like to be able to track the website for the campaign so I can see SEO gains / losses and increases / decreases in indexing.
Getting Started | | DrainKing0 -
Moz can't crawl my site.
Moz cannot carry out the site crawl on my online shop. Not really sure what the issue is, it has no problem getting onto my site when you use www. before the address, but it needs to be able to access bluerinsevintage.co.uk Stuck as what to do, we are a shopify store. Anyone else had this problem, or know what i need to change so they can crawl the site? thjis is the page they are getting when trying to get on bluerinsevintage.co.uk but if they use www.bluerinsevintage.co.uk the site comes up. Adam
Getting Started | | bluerinsevintage0 -
Https not sticking on competitor URL
One of our competitors uses Https:// on thier search results and when we add and save it, it removes the s - we know their site ranks on specific keywords, we know this from manually searching before we got moz but moz is not showing them because it is looking for http:// not https:// How can i fix this. Also - when we first set it up, we did not add the www. and our analytics are setup as such so we show no traffic on our dashboard, but it wont let me edit to change it. Thanks
Getting Started | | HansSchlamadinger
Tom1 -
Campaign.crawl-seed.bad-response
I am trying to set up a new campaign for a website, but I keep getting this error message... campaign.crawl-seed.bad-response 😞 I have no idea what the problem is. Can you tell me what I am suppose to do to fix this? The URL I am trying to set up is www.aboutplcs.com
Getting Started | | ChadC0 -
MOZ Starter Crawl not happeneing
Hi I added a new site 48hours + ago and the starter crawler has not even begun collecting data. Any help would be appreciated. cheers Isaac
Getting Started | | sodafizz0 -
Is there a way to force re-crawl?
1- Can we force the system to recrawl a website without having to wait for the scheduled date? 2- Please forgive me as I am new to this, but is "Moz Analytics" supposed to replace the "Pro"? Thank you!
Getting Started | | BlackTreeIT0 -
Cant download my crawl csv
When I click on the [download csv] in my crawl campaign section nothing happens.
Getting Started | | digitalmedialounge0 -
How do you "Moz Crawl" a website? Newbie...
Hi everyone; I've used Screaming Frog in the past and it's simple: you enter the URL to the box and click "start" and.. voila: as the button says, the crawling starts. I've had the Pro version of Moz for a while now and haven't really 'done' anything with it. I'd like to crawl a website and thought it would be as easy as it's always been with Screaming Frog... but, for some reason, I can't find the 'way' to do it. I find it really frustrating especially cause I feel like an idiot going around in circles thinking I'm missing something really obvious... Until I realised the only solution was to ask here! So... how in the world do you crawl a website using Moz tools? (Pro version) Thanks!
Getting Started | | patrihernandez1