We recently switched from HTTP to HTTPS and we are having crawling issues!
-
We switched our website from HTTP to HTTPS and we started to get an email from Moz about the robots.txt being unable to crawl our website. The website is hosted through wordpress but we haven't had any issues until we switched. We have no idea what to do or even what the problem is! If you have had a similar problem and fixed it, we need your help! Thank you.
-
I know this is an old thread, but we are still having the same problem. I finally got around to sending a note to flywheel about this problem and it came back that everything is fine. I am not sure what to do here? It's on a shared hosted, so I don't have console\audit log access, however Flywheel is one of the best wordpress hosting companies out there (only thing they do).
As far as accessing the robots.txt file, I can go directly to it without any problems?
https://southernil.com/robots.txt -
Hi there!
Thanks so much for reaching out! I'm sorry you're having trouble!
I took a look at your crawl data and your site to see if I could figure out the issue. When I first tried to access your robots.txt file from a browser, it returned an error saying there were too many redirects in place. I checked to see what our crawler was receiving from your server and looks like it keeps being served a 301 redirect which points back to itself. However, when I tried to access the file from a browser a bit later, it loaded without a problem. I'm wondering if you can check your server logs to see what your server is sending back to our crawler Rogerbot?
If you could send any further info over to help@moz.com, that would be great! That way we can do some more digging and see what's going on.
Looking forward to hearing from you!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can't Crawl Site - but deducting crawls.
Why am I being deducted crawls if MOZ keeps telling me that it can't crawl my site?
Getting Started | | BloggyMoms1 -
Keyword List issue
Hi, I have created a keyword list with 500 phrases and your software has been set as "Gathering Data" for hours now. I have closed, re-opened, refreshed multiple times and still nothing. Does it really take that long or is there some sort of issue?
Getting Started | | Crowleymjc0 -
HTTP or HTTPs for competitor URLs?
When setting up my competitors, should I use their http or https URLs? Thank you!
Getting Started | | Erik_Holm1 -
Got a problem in using MOZ Crawl test
Hello,
Getting Started | | turkeyanaclinic
Guys i need help as i'm getting this message "**Moz was unable to crawl your site on Dec 26, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster."
After i made a Campaign i'm getting this message but after i created new campaign it crawls well
can you help me to edit the old campaign ? Regards.0 -
901 error code showing url back to back in crawl
Hi Everyone, I'm absolutely dumbfounded about this 901 issue (showing pages with our url back to back). Our site is hosted on Big Commerce: https://www.santabarbarachocolate.com When I look for these pages being crawled I don't find them. I've called BC for help and I can't seem to find a solution or where to turn as to how to fix the issue at hand or even if it matters. Please see below what the Moz crawl shows. Could this be related to Yotpo or some app we have running? Or does this even matter and does it have any influence on rank? Do you have recommendations or ideas? Thanks so much. Pages with Crawl Attempt Error as of Mar 3 URL Page Authority Linking Root Domains Status Code | Error Code 901: DNS Errors Prevented Crawler from Resolving Hostname http://www.santabarbarachocolate.comhttp/www.santabarbarachocolate.com/100-percent-pure-cacao-unsweetened-baking-chocolate -- -- 901 Error Code 901: DNS Errors Prevented Crawler from Resolving Hostname http://www.santabarbarachocolate.comhttp/www.santabarbarachocolate.com/buy-wholesale-bulk-chocolate -- -- 901 Error Code 901: DNS Errors Prevented Crawler from Resolving Hostname http://www.santabarbarachocolate.comhttp/www.santabarbarachocolate.com/organic-chocolate-wholesale | -- | -- | 901 |
Getting Started | | santabarbarachocolate0 -
Mozbot Can Not Crawl Entire Domain
I'm trying to crawl Redken.com in Moz Analytics and the Search Diagnostics is only crawling 4 pages. The domain uses a "select your country" the first time you visit, and it seems as though the bot is not getting beyond that (aka, not clicking on "USA") and is therefore not crawling the rest of the domain. There is no country specific URL other than redken.com. I've tried entering both "redken.com" and "www.redken.com" as the URL, but no luck. Any tips?
Getting Started | | LabeliumUSA0 -
Https not sticking on competitor URL
One of our competitors uses Https:// on thier search results and when we add and save it, it removes the s - we know their site ranks on specific keywords, we know this from manually searching before we got moz but moz is not showing them because it is looking for http:// not https:// How can i fix this. Also - when we first set it up, we did not add the www. and our analytics are setup as such so we show no traffic on our dashboard, but it wont let me edit to change it. Thanks
Getting Started | | HansSchlamadinger
Tom1 -
Is there a way to force re-crawl?
1- Can we force the system to recrawl a website without having to wait for the scheduled date? 2- Please forgive me as I am new to this, but is "Moz Analytics" supposed to replace the "Pro"? Thank you!
Getting Started | | BlackTreeIT0