How do can the crawler not access my robots.txt file but have 0 crawler issues?
-
So I'm getting this errorOur crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.https://www.evernote.com/l/ADOmJ5AG3A1OPZZ2wr_ETiU2dDrejywnZ8kHowever, Moz is saying I have 0 Crawler issues. Have I hit an edge case? What can I do to rectify this situation? I'm looking at my robots.txt file here: http://www.dateideas.net/robots.txt however, I don't see anything that woudl specifically get in the way.I'm trying to build a helpful resource from this domain, and getting zero organic traffic, and I have a sinking suspicion this might be the main culprit.I appreciate your help!Thanks!
-
Hey there! Tawny from Moz's Help Team here.
It looks like your site's robots.txt file is returning some errors to our tools. When I try to visit the robots.txt file from the root domain, which is where our crawler starts, I get a warning that the DNS address can't be found: https://www.screencast.com/t/ezfsiyVso4B9 That same file is returning a 503 error to our crawler: https://www.screencast.com/t/ROlNo8AQz
That robots.txt file doesn't redirect anywhere, so you may want to consider putting in a redirect there to your robots.txt file at http://www.dateideas.net/robots.txt.
The reason you're seeing 0 issues reported is that we weren't able to reach your robots.txt file, so we stopped crawling and didn't have any issues to report.
I would speak to your web developer or whoever manages your site for you about making sure that your robots.txt file is fully accessible to our crawlers and can be reached in a browser.
I hope this helps! If you've still got questions, feel free to shoot us a note at help@moz.com and we'll do our best to sort everything out with you!
-
This just an advice that worked for me in the past for the same issue and of course is not in the user guide. Simple a delete the website or project from Moz Pro and create a new one for the same domain, I dont know why but all the errors and Issues disappeared.
-
Well technically if it can't crawl your site, it won't be able to find any issues...
I think this may be an error with the Moz crawler rather than any crawlers - checking with site:dateideas.net pulls up some results, although they seem to be tag and archive pages at the moment, which I'd be looking to noindex to avoid duplicate results in the future... You can always check Google's crawling by making sure you've registered your site with Google Search Console, and I'd suggest also making sure you've submitted your sitemap.
I have had issues in the past with the Moz bot struggling to crawl on specific hosts, so it might be that... The new crawl seems to work fine for me at the moment, but is it possible your site was down at the moment Moz was crawling it? You can always request a new crawl and monitor your site uptime to see if there's an issue...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can i fix? some information is wrong on the report.
https://moz.com/researchtools/ose/spam-analysis/flags?subdomain=bassinotary.com hi please advice. please see the link to my spam report. some information is wrong. like no email address or social profile link found I have already had links on that page and email and address and schema.org is in place. please help thanks.
Moz Bar | | grbassi0 -
How long after you create a campaign can you see a monthly view in dashboard?
I created a campaign on June 13th, 2017 and It's been about a month & two weeks and I'm still only seeing 'weekly' in my dashboard. I have tried searching how long it takes to be able to view monthly but I haven't found much information. Any help is appreciated! sXEAC8k.png
Moz Bar | | mkretsinger0 -
How can I find duplicate pages from a Moz Crawl?
We have many duplicate pages that show up on the Moz Crawl, and we're trying to fix these but it's very difficult because I can't see a way to isolate the code where the duplicate is found. For instance, http://experiencemission.org/immersion/ is one of our main pages, and the crawl shows one duplicate of http://experiencemission.org/immersion. It appears that one of our staff manually edited the source code in one of our pages but forgot the trailing slash. This would be an easy fix but the problem is that this page is linked to internally on our website 2423 times, so it's next to impossible to find the code that is incorrect. We have many other pages with this same basic problem. We know we have duplicates, but it's next to impossible to isolate them. So my question is this: When viewing the Moz Crawl data is there any way to see where a specific duplicate page link is located on our website? Thanks for any and all help!
Moz Bar | | expmission0 -
MOZ crawler has been finding a lot of 803 and 804 errors
During last 3 weeks MOZ crawler has been finding a lot of 803 and 804 errors. Meanwhile all pages seem to be working fine. What could cause it?
Moz Bar | | Paruyr0 -
Meta Description issue
Hi Guys, I'm going through my campaign report and i seem to have an excessive amount of missing Meta descriptions. How ever when i check the backend the descriptions seem to be present http://www.health2000.co.nz/shop/allergies/lung-zone/p5152/c55 Can any one explain to me why this is occurring?
Moz Bar | | Hardley1110 -
Can you adjust the timeframe from weekly to monthly in the Dashboard Overview report?
I am looking to adjust the metrics displayed in the dashboard Overview report and the other subsequent reports to display a monthly time frame instead of a weekly time frame. I see the pull down bar that looks like I should be able to adjust the time frame but only weekly is given as an option. I do not know if this is specific to my account, that is, I am running on a subscription that only allows for weekly metrics. Thank you, Jon
Moz Bar | | bcmull0 -
My 301 Error and Duplicate Title Content Issue is Growing !
When i redirect some of my page - it shows error. not redirecting and i made this 3-4 months before, no effect. All Errors under each category make me gone sick.
Moz Bar | | Esaky0