Moz unable to crawl my Zenfolio website
-
Hey guys,
I am attempting to optimize a website for my wife's business but Moz is unable to crawl it. Zenfolio is the web hosting service (she is a photographer).
The error message is:
**Moz was unable to crawl your site on Apr 1, 2019. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. Read our troubleshooting guide.
I did read the troubleshooting guide but nothing worked.
My robots.txt file disallows a few bots, but not roger bot. Anyone have any idea what is going on? Or do I need to request server logs from Zenfolio?
Thanks
-
It appears that Zenfolio really doesn't want Moz crawling their site for some reason. They want me to jump through hoops to get it to work and, being an ultra small business, cant justify the effort to get it working.
Guess I could always migrate to another web host.
Thanks for the replies.
-
Hi OP.
Hopefully you got in touch with Dave and the Moz team, and you have got your answer!
I haven't got a direct answer but from experience of Moz and other crawler softwares, my money would be on a temporary issue with your server being unresponsive.
As per the "Moz Can't Crawl your site guide, as it states, if it can't read your robots.txt file, then "Roger" won't crawl your site.
This is my simplest explanation I have
Hopefully it's resolved soon!
P.S
I looked at Zenfolio, if it's a self-hosted CMS system, it might be worth checking they haven't got any special rules in place to block Moz and other popular crawlers
-
Hey! Dave here from the Help Team
Can you write into us at help@moz.com with the site that is having crawl issues? We can take a look and help you out.
thanks!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz tried to charge our credit card
Hello, We noticed that MOZ tried to charge our credit card, while we are on Trial period until today. We decided to cancel the subscription (until our budget grows and come back). The tool is great, but it currently is too expensive for our business. We enjoyed the trial period and will surely come back when our business can afford it!! Thanks!
Getting Started | | intenseed0 -
What does Location National means in Moz Pro Campaigns?
I understand the concept of locations in Moz Pro Campaigns. What does National exactly means? Is it like globally for a United States Google Search Rank or what? Please explain.
Getting Started | | ds9.tech1 -
901 error code showing url back to back in crawl
Hi Everyone, I'm absolutely dumbfounded about this 901 issue (showing pages with our url back to back). Our site is hosted on Big Commerce: https://www.santabarbarachocolate.com When I look for these pages being crawled I don't find them. I've called BC for help and I can't seem to find a solution or where to turn as to how to fix the issue at hand or even if it matters. Please see below what the Moz crawl shows. Could this be related to Yotpo or some app we have running? Or does this even matter and does it have any influence on rank? Do you have recommendations or ideas? Thanks so much. Pages with Crawl Attempt Error as of Mar 3 URL Page Authority Linking Root Domains Status Code | Error Code 901: DNS Errors Prevented Crawler from Resolving Hostname http://www.santabarbarachocolate.comhttp/www.santabarbarachocolate.com/100-percent-pure-cacao-unsweetened-baking-chocolate -- -- 901 Error Code 901: DNS Errors Prevented Crawler from Resolving Hostname http://www.santabarbarachocolate.comhttp/www.santabarbarachocolate.com/buy-wholesale-bulk-chocolate -- -- 901 Error Code 901: DNS Errors Prevented Crawler from Resolving Hostname http://www.santabarbarachocolate.comhttp/www.santabarbarachocolate.com/organic-chocolate-wholesale | -- | -- | 901 |
Getting Started | | santabarbarachocolate0 -
Moz scraper
How often do you Moz do whatever it is they do for me to get up-to-date data?
Getting Started | | infinety0 -
Moz for charities and non-profits
Hi, I noticed an earlier thread on this topic that now seems dead so I decided to start a new one: https://moz.com/community/q/non-profit-rates We're a non-profit organisation based outside of the US and we were wondering if MOZ's non-profit program can be extended out? We're fully registered and audited in our home country and can provide all the necessary documentary proof if needed.
Getting Started | | OBW_Noel1 -
When connecting your Google Analytics account to your Moz account, Moz provides a drop-down of accounts & UA codes to select from. Where is Moz pulling those selection options from?
The list of Google Analytics accounts and UA codes listed are outdated. How can I update them to reflect the most up-to-date accounts (& UA codes associated with those accounts)?
Getting Started | | SearchParty0 -
Trending Bugs in Moz Analytics
The FAQ’s We hope we can help! Below are the trending issues at the moment. If you don’t see your question addressed or need further help, send us a message at http://moz.com/help/contact. Our Help Team gumshoes will investigate your issue and respond shortly. Where did my Campaigns Go?
Getting Started | | Abe_Schmidt
Whenever your account is suspended, your campaigns are archived. The good news is that reviving archived accounts is really simple. Go to your Campaigns section: http://pro.moz.com/campaigns
At the top of the page there is a "Archived Campaigns" Tab, give it a click.
From here, you should easily be able to "activate" your campaigns. My Google Analytics profile won’t stay connected to my campaign and/or I am missing profiles in my GA Settings. Google goes through a process called Oauth when it comes to authorizing access of services that are linked to your Google account. Under any single Google account, there are about 20 tokens account wide. Those tokens are used to provide access for a variety of services from Gmail to apps on your phone. Once you hit the limit, which is 20 for most users, the system automatically revokes the oldest token to provide a new one. I'm not certain if this is causing the issue on your account, but it is a great place to start troubleshooting. You may be able to correct the issue by manually revoking your tokens to make room for new ones on your account and then reconnecting the account to Moz. Here are the instructions on how to do that: Follow this link https://accounts.google.com/b/0/IssuedAuthSubTokens. This page displays the current OAuth tokens you are using. Once you reach the page, simply press the revoke button (illustrated on this screenshot: http://screencast.com/t/vjh3KrjRRIe) for services that you are not using right now. Once you are done with, that simply go back to your campaign settings, disconnect your GA profile and reconnect. The below process may also fix this issue for you. Head to your campaign settings page on your overview page. Disconnect your Google Analytics connection. Go back to your settings page and click on "connect account." Please make sure you log into the correct GA account. 🙂 Hurray! This should let you grab the most current traffic data! Oops! try refreshing page! I can’t access any of my Campaign data.
This issue is normally machine specific and can be a bit complex. An individual forum has been created to address this: https://seomoz.zendesk.com/entries/28203486-Oops-Try-Refreshing-the-Page-Error-showing-on-all-pages- Unable to retrieve historical ranking CSVs. Some users did not receive their requested Historical Rankings CSVs. This has mostly been affected by a change or update to your competitors after a campaign has been setup for a while. Please send your request to retrieve this data with http://moz.com/help/contact.8 -
How long does it usually take Moz to populate information for a new Web site?
We recently launched (9/13/2013) an e-commerce Website and added the campaign to SEO MOZ. Week after week the Domain Rank is 1 and none of our keyword stats or link stats are populated. We have another Moz campaign that posts weekly updates and is doing extremely well. I'm just wondering how long it usually takes Moz to start populating all the analysis stats? I'm also wondering if there might be a campaign setting buried somewhere that I need to enable or maybe it just takes more than 5 weeks? Any insights would be much appreciated. Here's the new URL we need to track with MOZ: http://www.imsportshq.com
Getting Started | | Tripper0