When I crawl my site On Moz it says it can't access the robots.txt file, but crawl is fine on SEM Rush - Anyone know any reason for this?
-
Hi guys,
When I try to run a site crawl on Moz it returns an error saying that it has failed due to an error with the robots.txt file.
However, my site can be crawled by SEM Rush with no mention of problems with roots.txt file issues.
My developer has looked into it and insists their is no problem with my robots.txt and I've tried the Moz crawl at least 6 times over an 8 week period.
Has anyone ever seen such a large discrepancy between Moz and SEM Rush or have any ideas why Moz has this issue with my site??
TIA everyone
-
Hi there!
Thanks so much for the great question! I'm so sorry you're having trouble getting your site crawled. Without knowing the exact site you're working with I can't say for sure what's going on but I can offer some suggestions that may help. If you're seeing the crawl is successful with other tools, there may be a bot-specific setting or directive that is banning our crawler, rogerbot, from your site. This may be coming directly from your server or it may be listed in your robots.txt file.
I have a troubleshooting guide here that may help- it outlines some other common issues that can keep rogerbot from being able to move forward with the crawl.
If you're still having trouble, please feel free to send an email on over to help@moz.com with the site you're having trouble with. That way we can take a look and see what's going on!
-
I also had this issue today on my site. Haven’t tried Semrush but got the same error that you are describing when I used Moz
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I want to increase the DA of my Site
Dear all i have established a new site and want to increase its DA fast. So i need your suggestions in this regard. This newly established blog is my first independant project so i want to make it perfect.
Getting Started | | hamza52520 -
Why Moz DA of my website not increasing?
Hi, I have two queries which are related to Moz DA and Moz Spam Score. 1. I have created my website 6 months ago and also do proper SEO of my site. Now according to Ahrefs my DR is 47 and also shows me some organic traffic, Organic Keyword of my site. But when I've checked my site in Moz.com it shows me the DA is only 1 and also there is no organic keywords and no backlinks. Why my DA is not increasing and stuck in 1 position? When will DA increase and please guide me how I can Improve DA of my website? URL: https://www.fasterapk.com 2. My second query is, what is the Moz Spam Score? Is it important for a website and which factors Moz used to create a Spam Score of any website? Experts guide needed. Thanks, overview?site=https%3A%2F%2Ffasterapk.com⌖=domain
Getting Started | | dekho870 -
We recently switched from HTTP to HTTPS and we are having crawling issues!
We switched our website from HTTP to HTTPS and we started to get an email from Moz about the robots.txt being unable to crawl our website. The website is hosted through wordpress but we haven't had any issues until we switched. We have no idea what to do or even what the problem is! If you have had a similar problem and fixed it, we need your help! Thank you.
Getting Started | | DrInfinity0 -
My page can not be crawled
Hi all,Hope you could you help me here.I just seen this message but I don't know how to fix it?"Your page redirects or links to a page that is outside of the scope of your campaign settings. Your campaign is limited to pages with _____ in the URL path, which prevents us from crawling through the redirect or the links on your page. To enable a full crawl of your site, you may need to create a new campaign with a broader scope, adjust your redirects, or add links to other pages that include ig.com/de. Typically errors like this should be investigated and fixed by the site webmaster."Any ideas about how should I fix it?Thanks
Getting Started | | lauracelada23100 -
MOZ Guru Needed for Freelance Work
Hello -- I'm new to Moz and wanted to know if any community members were available for freelance work to help me fine-tune the setup of the Moz account for one of my clients. This could lead to additional work for other clients of my advertising agency.
Getting Started | | KRT0 -
Where is my access id?
Hi, i am using a 3rd party wordpress plugin (WPMU DEV - Infinite SEO). I've got a trial account and the plugin is asking me for: Access ID Secret Key Where can i find these? much appreciated graham
Getting Started | | aguyiknow1 -
Daily crawl reports, are they wasting my time?
I am relatively new here, I have 5 campaigns. I get new crawl complete reports almost every day for all of them. Wow great, except when I check the reports nothing has changed. Even if I have gone in and changed things or fixed errors, the same ones are still there and takes 4-7 days for that work to show up. Everytime I get one of these reports I am opening them up going through and not seeing the changes I implemented the previous days before. I'll spend 20-30 minutes going over these and checking details. So the question is, Are these reports wasting my time? Are they actually new reports or am I just getting spammed repeat notices everyday?
Getting Started | | RandyFriesen0 -
Lost since the change over to Moz
I was recently lead to a series of videos which looked to be quite helpful here http://moz.com/help/pro/introduction But is seems with the change over from SEO Moz to Moz, nothing is in the same place. Should I bother watching them or wait for new ones? I'm particularly interested in SEO to raise our rank.
Getting Started | | AliciaMarie0