How do can the crawler not access my robots.txt file but have 0 crawler issues?
-
So I'm getting this errorOur crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.https://www.evernote.com/l/ADOmJ5AG3A1OPZZ2wr_ETiU2dDrejywnZ8kHowever, Moz is saying I have 0 Crawler issues. Have I hit an edge case? What can I do to rectify this situation? I'm looking at my robots.txt file here: http://www.dateideas.net/robots.txt however, I don't see anything that woudl specifically get in the way.I'm trying to build a helpful resource from this domain, and getting zero organic traffic, and I have a sinking suspicion this might be the main culprit.I appreciate your help!Thanks!
-
Hey there! Tawny from Moz's Help Team here.
It looks like your site's robots.txt file is returning some errors to our tools. When I try to visit the robots.txt file from the root domain, which is where our crawler starts, I get a warning that the DNS address can't be found: https://www.screencast.com/t/ezfsiyVso4B9 That same file is returning a 503 error to our crawler: https://www.screencast.com/t/ROlNo8AQz
That robots.txt file doesn't redirect anywhere, so you may want to consider putting in a redirect there to your robots.txt file at http://www.dateideas.net/robots.txt.
The reason you're seeing 0 issues reported is that we weren't able to reach your robots.txt file, so we stopped crawling and didn't have any issues to report.
I would speak to your web developer or whoever manages your site for you about making sure that your robots.txt file is fully accessible to our crawlers and can be reached in a browser.
I hope this helps! If you've still got questions, feel free to shoot us a note at help@moz.com and we'll do our best to sort everything out with you!
-
This just an advice that worked for me in the past for the same issue and of course is not in the user guide. Simple a delete the website or project from Moz Pro and create a new one for the same domain, I dont know why but all the errors and Issues disappeared.
-
Well technically if it can't crawl your site, it won't be able to find any issues...
I think this may be an error with the Moz crawler rather than any crawlers - checking with site:dateideas.net pulls up some results, although they seem to be tag and archive pages at the moment, which I'd be looking to noindex to avoid duplicate results in the future... You can always check Google's crawling by making sure you've registered your site with Google Search Console, and I'd suggest also making sure you've submitted your sitemap.
I have had issues in the past with the Moz bot struggling to crawl on specific hosts, so it might be that... The new crawl seems to work fine for me at the moment, but is it possible your site was down at the moment Moz was crawling it? You can always request a new crawl and monitor your site uptime to see if there's an issue...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I can't find multiple titles
Hello, Moz showed a lot of multiple titles on our website but, when I go to fix them, there isn't any multiple titles. I was wondering if anyone knew where these issues could be coming from? Many thanks, Olivia
Moz Bar | | SigneerHFS0 -
"New" issues not previously found being shown?
I'm not sure what logic Moz is using for its reporting of Site Crawl issues, but it appears to be pretty flawed (unless I'm missing something, which is possible). I've got a client site that has been in Moz for about 6 months now. Every time the crawler runs, the same number of pages are reported as having been crawled. However I'm consistently getting "New Issues" reported that should have been reported during previous crawls. Example: A redirect chain was reported several month ago. The referring URL was the homepage of the website, and we tracked it down to an old link in the header. This was fixed, marked as resolved, and the issue was not shown on the next crawl. Several weeks later, the same issue was reported for a different page on the website - a page which has existed since 2014 and was already crawled many times. Again, we fixed. Fast-forward to the report that just ran on 12/1 and we have the same issue reported, for a different page, which has also existed for years and has been previously crawled. It's very hard to explain to a client "this item you are seeing has been resolved", only to have it continually crop back up in future reports. Note this is not limited to redirect chains - that's just an example. I'm seeing this for other items such as missing canonicals, duplicate titles, etc.
Moz Bar | | RucksackDigital0 -
Issue with Keyword List Export
I am having an issue with the Keyword list export. I exported a keyword list, then went back in a made changes to the list (deleted keywords, changed priority, etc). Then when I re-export the list it still downloads the original list. For instance, I had a keyword list of 120 keywords. I exported it. Then I reduced the list to 30 keywords. Now when I export that list it is still the original 120 keywords. Any one else having this issue? Is there a way to fix this?
Moz Bar | | Sundaram1 -
Why do new issues occur when there have been no updates to the website?
I am unsure on how new issues are created when crawling the site from one crawl to the next. I had assumed the number of issues would remain static if nothing was done from one crawl to the next. Can someone explain how new/additional issues are added from each crawl, even without updates being made? Thanks in advance, Christopher
Moz Bar | | debb0 -
Www and non www / duplicate content / redirects / www resolve issue
I am not getting docked for these specific errors, but I am getting docked for 1 page has a WWW resolve issue and 1 wrong URL in the sitemap... (SEM Rush) but when I use moz, it's not showing any issues. So I have these things set up so far: In .htaccess i have a command that removes the www. 301 redirect from www version to the non www (homepage) canonical on index.html pointing to non www version, I also set up a canonical tag for each page on the site search console with non www, www, https www, https non www all set to non www preference. Also, when I fetch the www version in google search console it says it's being 301 redirected to non www version which is basically what I want.Is there anything that i'm missing? These errors on SEM Rush are giving me anxiety lol.
Moz Bar | | donnieath1 -
How can I print out the keyword ranking chart?
I have been working diligently on these keywords. Not a lot of them but trying to get very specific. I have one that has traditionally been very high for our site and in one day it dropped from 2 to past 50+ when I reviewed the history I was unable to find any place to print or download the graph for the history or any place to print a report for our keyword rankings.
Moz Bar | | signcreator0 -
Why is 410 (Gone) being classed as a high priority issue in crawl diagnostics?
Are high priority issues have suddenly soared by over 100 because Moz is classing 410s as high priority.
Moz Bar | | Melissabraz
Google doesn't class these as so serious, so we were wondering if anyone knows why Mos does?0 -
Rel Can notice issue on my SEOMoz reporting
Need some help understanding this report... I have 17 notices for Rel Can on my campaign. Then, it lists all the links. But what is this report actually telling me? Is it telling me that Rel Can's are listed on these pages? The are all blog posts...our blog was redirected when the site was recently rebuilt. I just need to understand what the report is really telling me to do/not do. Or is it ok to ignore this "notice"?
Moz Bar | | cschwartzel0