When I crawl my site On Moz it says it can't access the robots.txt file, but crawl is fine on SEM Rush - Anyone know any reason for this?
-
Hi guys,
When I try to run a site crawl on Moz it returns an error saying that it has failed due to an error with the robots.txt file.
However, my site can be crawled by SEM Rush with no mention of problems with roots.txt file issues.
My developer has looked into it and insists their is no problem with my robots.txt and I've tried the Moz crawl at least 6 times over an 8 week period.
Has anyone ever seen such a large discrepancy between Moz and SEM Rush or have any ideas why Moz has this issue with my site??
TIA everyone
-
Hi there!
Thanks so much for the great question! I'm so sorry you're having trouble getting your site crawled. Without knowing the exact site you're working with I can't say for sure what's going on but I can offer some suggestions that may help. If you're seeing the crawl is successful with other tools, there may be a bot-specific setting or directive that is banning our crawler, rogerbot, from your site. This may be coming directly from your server or it may be listed in your robots.txt file.
I have a troubleshooting guide here that may help- it outlines some other common issues that can keep rogerbot from being able to move forward with the crawl.
If you're still having trouble, please feel free to send an email on over to help@moz.com with the site you're having trouble with. That way we can take a look and see what's going on!
-
I also had this issue today on my site. Haven’t tried Semrush but got the same error that you are describing when I used Moz
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why Moz DA of my website not increasing?
Hi, I have two queries which are related to Moz DA and Moz Spam Score. 1. I have created my website 6 months ago and also do proper SEO of my site. Now according to Ahrefs my DR is 47 and also shows me some organic traffic, Organic Keyword of my site. But when I've checked my site in Moz.com it shows me the DA is only 1 and also there is no organic keywords and no backlinks. Why my DA is not increasing and stuck in 1 position? When will DA increase and please guide me how I can Improve DA of my website? URL: https://www.fasterapk.com 2. My second query is, what is the Moz Spam Score? Is it important for a website and which factors Moz used to create a Spam Score of any website? Experts guide needed. Thanks, overview?site=https%3A%2F%2Ffasterapk.com⌖=domain
Getting Started | | dekho870 -
I’d like to set up a Moz campaign that crawls just the primary website, not subdomains
Is this something you could help with? (it either bombs or crawls everything, so I assume I’m missing something in the campaign settings, or it’s just not possible.
Getting Started | | blueprintatl0 -
Can I view Domain Authority stats for longer than the previous 12 months?
When you view the DA chart, it goes back 12 months. There is no apparent way to see any data before that. I searched the Q&A but see no other similar questions. Any insight?
Getting Started | | cptutty0 -
Why does the moz bar link count differ from the moz link analysis page
Hi all, Why does the Moz Bar show a different link count from the Moz link analysis page? For example, when I check the SERP below, for the first result, the bar shows 936 page links from 4 RDs. But when I check out the link analysis page, it tells me there are just 141 page links from 4 RDs. What gives? For the second entry, the bar shows 6 page links from 0 RDs. Not sure how that's possible. Can anyone explain these things. Thanks! Andy Reviewed SERP: https://www.google.com/search?source=hp&ei=hhRVW5yyH5C60PEP-_-isAw&q=mountain+bike+trails+near+me&oq=mountain+bike+trails+near+me&gs_l=psy-ab.3..0l7j0i22i30k1l3.645.4654.0.5322.28.19.0.8.8.0.243.2920.3j11j5.19.0....0...1c.1.64.psy-ab..2.26.2961...0i131k1.0.C4lAxLkGgH0
Getting Started | | AndyKubrin0 -
I haven't used Moz and you've been billing me. Please refund thanks
I signed on for a free trial and found out today that you've billed me in June and July. Check my history as this is only the second time i've logged in and it's just to request you refund.
Getting Started | | Completebanker0 -
Moz Staff: History + Watch List
Hi All, I have been going crazy trying to find a thread I read last week ... I think!
Getting Started | | Mark_Ch
Several search attempts later, no closer to finding the thread in question. A genius moment or simple frustration got me thinking. How cool would it be to have the following features on MOZ Q&A Forum. Watch List
1] Check off any threads that you would like to monitor over time. Should you stumble upon a thread of value then simply add to watch list and view it's progress over time. No searching for the thread in question. History
2] Keep a history off all threads viewed over a period of time. Thanks Mark0 -
How do I update the crawl issues & Notifications?
I have a list of errors, most relating to missing meta descriptions. I have added a meta description to a page, visited the site and viewed the source, and the meta description is now there. When I go to analyze issues, the report it gives back for the link contains the same missing meta description as previously. How do I get it to update and realize the issue has been fixed?
Getting Started | | ETGg0 -
Getting Redirect Loops in MOZ using Chrome
Been getting bizarre Redirect Loops from Chrome after I log-in to MOZ. Has anyone had something like this happen? I've tried clearing cache, rebooting, etc. but no luck. Thanks in advance!
Getting Started | | danny.wood1