Moz unable to crawl my Zenfolio website
-
Hey guys,
I am attempting to optimize a website for my wife's business but Moz is unable to crawl it. Zenfolio is the web hosting service (she is a photographer).
The error message is:
**Moz was unable to crawl your site on Apr 1, 2019. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. Read our troubleshooting guide.
I did read the troubleshooting guide but nothing worked.
My robots.txt file disallows a few bots, but not roger bot. Anyone have any idea what is going on? Or do I need to request server logs from Zenfolio?
Thanks
-
It appears that Zenfolio really doesn't want Moz crawling their site for some reason. They want me to jump through hoops to get it to work and, being an ultra small business, cant justify the effort to get it working.
Guess I could always migrate to another web host.
Thanks for the replies.
-
Hi OP.
Hopefully you got in touch with Dave and the Moz team, and you have got your answer!
I haven't got a direct answer but from experience of Moz and other crawler softwares, my money would be on a temporary issue with your server being unresponsive.
As per the "Moz Can't Crawl your site guide, as it states, if it can't read your robots.txt file, then "Roger" won't crawl your site.
This is my simplest explanation I have
Hopefully it's resolved soon!
P.S
I looked at Zenfolio, if it's a self-hosted CMS system, it might be worth checking they haven't got any special rules in place to block Moz and other popular crawlers
-
Hey! Dave here from the Help Team
Can you write into us at help@moz.com with the site that is having crawl issues? We can take a look and help you out.
thanks!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best use of Moz Pro
Hello, I want to ask what are the benefits of Moz Pro other than finding low competitive keywords? I want to know the best use of Moz Pro for betterment of my site (chicken grill recipe). I want the best possible result. Guide me. Thanks.
Getting Started | | maxcharles0 -
Multiple campaigns for one website with multiple services?
Hi. Brand new to Moz Pro today. Question about whether to set up multiple campaigns for one website that promotes multiple services. I own a law firm and we have one main location, one website, 5 attorneys, and about 8-10 practice areas we promote. Each practice area has its own page and related content in the form of guides and blogs. Website is southernmarylandlaw.com. I only have this one website to track and promote, so I initially thought I only have one campaign. But I do want to track these separate practice areas separately. Each will have separate competitors, for instance. Each will need a separate set of keywords. People tend to search for a particular type of attorney, not just for the generic "lawyers." Should I use my 10 campaigns to set up a different campaign for each practice area we promote? Am I doing something wrong by tracking my one website 10 different times in my account? If I do not do this, then how do I possibly track the various practice areas and their separate keywords and competitors? Thanks for all your help and suggestions!
Getting Started | | tuckervc0 -
Moz scraper
How often do you Moz do whatever it is they do for me to get up-to-date data?
Getting Started | | infinety0 -
Multiple Google Analytics Accounts connected to the same MOZ subscription
Hi everyone, I have a quick question regarding setting up my MOZ account and it's
Getting Started | | Bernardo.Reguero
integration with a Google Analytics account: I'm not sure if I'm right but I think I can only set a single Google
Analytics account to do my analysis and then created several campaign. Or is there a possibility to set several Google Analytics accounts and creating several camapign to track diferente websites (connected to diferente Google Analytics accounts) ? Thanks! Cheers, Bernardo0 -
Why I can't add more campaign to my moz analytics?
As we know that being a standard pro member in Moz community, we have at least 5 campaigns to add and track & as well in new moz analytics. But when I just made http://www.norwoodgreen.in website campaign an archive to make active or add one of the old himachalpackagetour.com campaign which I was unable to do WHY WHY WHY? Let me know . .? Also do guide or train me How to use Moz analytics as I was not enough familiar with it as I am habitual with in Pro tools. Best,
Getting Started | | Futura
Teginder0 -
How to get moz to crawl a staging domain that is blocked by robots.txt
Is it possible to get Moz to do a crawl report on a domain blocked by robots.txt and actually display all errors instead of only one saying the domain was blocket in robots.txt? Anything i can add to robots.txt to make moz able to do the crawl report but still hinder google from crawling a staging domain?
Getting Started | | classifiedtech0