How do can the crawler not access my robots.txt file but have 0 crawler issues?
-
So I'm getting this errorOur crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.https://www.evernote.com/l/ADOmJ5AG3A1OPZZ2wr_ETiU2dDrejywnZ8kHowever, Moz is saying I have 0 Crawler issues. Have I hit an edge case? What can I do to rectify this situation? I'm looking at my robots.txt file here: http://www.dateideas.net/robots.txt however, I don't see anything that woudl specifically get in the way.I'm trying to build a helpful resource from this domain, and getting zero organic traffic, and I have a sinking suspicion this might be the main culprit.I appreciate your help!Thanks!
-
Hey there! Tawny from Moz's Help Team here.
It looks like your site's robots.txt file is returning some errors to our tools. When I try to visit the robots.txt file from the root domain, which is where our crawler starts, I get a warning that the DNS address can't be found: https://www.screencast.com/t/ezfsiyVso4B9 That same file is returning a 503 error to our crawler: https://www.screencast.com/t/ROlNo8AQz
That robots.txt file doesn't redirect anywhere, so you may want to consider putting in a redirect there to your robots.txt file at http://www.dateideas.net/robots.txt.
The reason you're seeing 0 issues reported is that we weren't able to reach your robots.txt file, so we stopped crawling and didn't have any issues to report.
I would speak to your web developer or whoever manages your site for you about making sure that your robots.txt file is fully accessible to our crawlers and can be reached in a browser.
I hope this helps! If you've still got questions, feel free to shoot us a note at help@moz.com and we'll do our best to sort everything out with you!
-
This just an advice that worked for me in the past for the same issue and of course is not in the user guide. Simple a delete the website or project from Moz Pro and create a new one for the same domain, I dont know why but all the errors and Issues disappeared.
-
Well technically if it can't crawl your site, it won't be able to find any issues...
I think this may be an error with the Moz crawler rather than any crawlers - checking with site:dateideas.net pulls up some results, although they seem to be tag and archive pages at the moment, which I'd be looking to noindex to avoid duplicate results in the future... You can always check Google's crawling by making sure you've registered your site with Google Search Console, and I'd suggest also making sure you've submitted your sitemap.
I have had issues in the past with the Moz bot struggling to crawl on specific hosts, so it might be that... The new crawl seems to work fine for me at the moment, but is it possible your site was down at the moment Moz was crawling it? You can always request a new crawl and monitor your site uptime to see if there's an issue...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
The factors considered in the new domain authority algorithm? On-site factors we can use to compare with competitors? Is having an NA as a spam score a bad thing?
Does anyone know the factors considered in the new domain authority algorithm other than spam score and complex distributions of links based on quality and traffic? Does anyone know of on-site factors we can use to compare with competitors to try and improve DA? Is having an NA as a spam score a bad thing?
Moz Bar | | CQMarketing0 -
Why do new issues occur when there have been no updates to the website?
I am unsure on how new issues are created when crawling the site from one crawl to the next. I had assumed the number of issues would remain static if nothing was done from one crawl to the next. Can someone explain how new/additional issues are added from each crawl, even without updates being made? Thanks in advance, Christopher
Moz Bar | | debb0 -
Can someone help me?
I need to use the competition analysis tool efficiently. I don't want to choose too big or too small competition, I am a startup affiliate website with 0 traffic. Is there guidelines for doing this?
Moz Bar | | hassan.houta0 -
Odd crawl test issues
Hi all, first post, be gentle... Just signed up for moz with the hope that it, and the learning will help me improve my web traffic. Have managed to get a bit of woe already with one of the sites we have added to the tool. I cannot get the crawl test to do any actual crawling. Ive tried to add the domain three times now but the initial of a few pages (the auto one when you add a domain to pro) will not work for me. Instead of getting a list of problems with the site, i have a list of 18 pages where it says 'Error Code 902: Network Errors Prevented Crawler from Contacting Server'. Being a little puzzled by this, i checked the site myself...no problems. I asked several people in different locations (and countries) to have a go, and no problems for them either. I ran the same site through Raven Tool site auditor and got some results. it crawled a few thousand pages. I ran the site through screaming frog as google bot user agent, and again no issues. I just tried the fetch as Gbot in WMT and all was fine there. I'm very puzzled then as to why moz is having issues with the site but everyone is happy with it. I know the homepage takes 7 seconds to load - caching is off at the moment while we tweak the design - but all the other pages (according to SF) take average of 0.72 seconds to load. The site is a magento one so we have a lengthy robots.txt but that is not causing problems for any of the other services. The robots txt is below. Google Image Crawler Setup User-agent: Googlebot-Image
Moz Bar | | Arropa
Disallow: Crawlers Setup User-agent: * Directories Disallow: /ajax/
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
#Disallow: /js/
#Disallow: /lib/
Disallow: /magento/
#Disallow: /media/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/
Disallow: /catalog/product
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
#Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/ Files Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt Paths (no clean URLs) #Disallow: /.js$
#Disallow: /.css$
Disallow: /.php$
Disallow: /?SID= Pagnation Disallow: /?dir=
Disallow: /&dir=
Disallow: /?mode=
Disallow: /&mode=
Disallow: /?order=
Disallow: /&order=
Disallow: /?p=
Disallow: /&p= If anyone has any suggestions then please i would welcome them, be it with the tool or my robots. As a side note, im aware that we are blocking the individual product pages. Too many products on the site at the moment (250k plus) which manufacturer default descriptions so we have blocked them and are working on getting the category pages and guides listed. In time we will rewrite the most popular products and unblock them as we go Many thanks Carl0 -
Moz crawl issues: All pages keep resolving to our "cookies not enabled" page
Upon running the Moz Pro site crawler, I noticed that I received quite a bit of duplicate titles along with 302 redirects (which is our site creating a temporary 302 to our "cookies not enabled" page). How would I get around the crawler being redirected to this page? I've never ran across this issue before, despite using the crawler with sites that use the same framework as the one thats affected. Any ideas?
Moz Bar | | responsivelabs0 -
Can you change date of monthly reports delivery ?
Hi Ive just set up a monthly report for a client but is scheduled to arrive BY the day they need it which is a bit close hence ideally would like to bring it forward by a few days, is this possible ? Cheers Dan
Moz Bar | | Dan-Lawrence0 -
Can Moz use canconical links to prevent notices about duplicate content issues?
if so how do we enable this - we've an average size site with a few hundred products but they appear in multiple categories, canonical url points to it's primary category (but a new page exists for each section... so for /cat-a/abc there will be another page cat-b/abc and again but the canonical points to cat-a always for that product) basically I see this kind of duplication error / notice as a false positive... help me
Moz Bar | | SEOAndy0 -
"Avoid Keyword Self-Cannibalization" - can't find the problem
Hi, I understand what this means (or at least I think I do!), but I can't find where the problem lies. The keyword is "fire warden training" and the url is http://www.tutis-fire.co.uk/fire-warden-training-courses/ If anyone could lend a helping hand, I'd appreciate it.
Moz Bar | | Gordon_Hall0