I'm getting a Crawl error 605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
-
The website is www.bigbluem.com and is a wordpress site.
I'm getting the following error:
605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
But what is weird is the domain it lists below that is http://None/BigBlueM.com
Any advice?
-
I can now resolve the www version of the site but not reach the root domain which will continue to return the 605 error so there is something about the root domain configuration that is blocking our bot. A workaround would be to create a new campaign for www.bigbluem.com instead of bigbluem.com
-
Thanks David!
I noticed that this morning it was showing the correct domain all of a sudden. Thanks for looking into that further.
I've made that change to the robots.txt file so go ahead and test when you can.
Thanks again!
-
Hello!
Sorry for the confusion. For your site there were two issues, one on our side with our crawler failing between August 21st - 29th crawls trying to reach http://None/yourdomain.com and the site not responding to robots.txt
The crawl issue has been resolved but for some reason your site is still blocking our user-agent.
Rogerbot can't follow allow directives well so you could try updating it with Disallow:
User-agent: Rogerbot
Disallow:Let me know if this helps! Once you make the change I can run a quick test to see if it will resolve.
-
Thanks!
I've made that change, although I still don't know why it would have the URL as http://None/BigBlueM.comThat's concerning and makes me think that it isn't crawling because it's trying to crawl that URL which doesn't exist.
Do you know if I have to wait another week for Moz to attempt a crawl or can I force that to see if it's working?
-
Thanks for piping in here! I will definitely rely more heavily on GWMT and will check out Screaming Frog SEO spider. Thanks!
-
I've consistently experienced problems with the Moz crawler - to the point of I no longer put much value into it.
I'm getting this error and nothing has changed in my robots.txt.
Instead, use GWMT and Screaming Frog SEO spider - that's all you need and does more than the Moz crawler.
-
I have noticed underneath rogerbot you have dissallow change it to
User-agent: rogerbot
Allow: /
Then let me know how you get on crawling the home page.
-
No problem, I will have a look into that issue for you now it does seem strange.
-
Hey! Thanks for the response.
I didn't have Disallow set up for the root folder at all, just for /wp-admin/.
I went ahead and added the User-agent: rogerbot
The one thing I am still concerned about is that Moz is saying "We were unable to access your homepage" and then has the URL http://None/BigBlueM.com
Why does it think that is my homepage? That seems weird and it isn't that way on any of my other sites that are set up in Moz.
-
In order to allow crawlers to to access the site, you would need to either remove the / after Disallow or change Disallow to Allow.
If you specifically want to allow the Moz crawler, you can insert the following directive above the current directive that is in the .htaccess
User-agent: rogerbot
Disallow: -
If you happen to figure out a solution before someone posts here, let me know what it is!
-
I am having the exact same problem, however Google webmaster tools is able to crawl the site just fine.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why is Moz Crawling More Pages Than My Site Actually Has?
Hi I have a site that only has 5k pages but Moz has crawled 50K pages on the site when I initiated the site crawl. I don't exactly know why Moz is reporting me back so many pages but I was wondering why this is and if any of you out in the Moz community know anything about this. Thanks
Moz Bar | | drewstorys0 -
Puzzled. High domain authority, but low page authority?
I have a question regarding how a page's authority can be low but the domain authority be high. I'm using the SERP report in moz to compare competing pages. The page I'm working on is an ecommerce category page. Moz says it has very low page authority, has been indexed for a little over a year, it's domain authority is a moderate 50, and has no root domains linking to it. It is more relevant than the other pages (has more products for sale that fit the keyword searched for), has a decent amount of copy on the page, but ranks around #10 in the search results. The competing pages are also ecommerce category pages. The 9 ranking higher have domain authority varying from 30-75, and 1 or 2 root domain links to them. Their number of products that match the keyword searched are all over the board, some have many matches, some have only 6 or 8 results. Some have a moderate amount of text on the page, some have very little text on the page. Yet they have page authority in the 30s-40s. What goes into page authority that I'm missing which would make these other pages have so much more authority when they are just slightly better in the other metrics listed above? What am I missing here?
Moz Bar | | K-WINTER0 -
Learn how to use Rank Tracker to analyze rank progress for any given keyword. Get your Daily SEO Fix!
Moz's Rank Tracker tool retrieves search engine rankings for pages and keywords, and stores them for easy comparison later. With it, you can analyze and track rank progress for a keyword or set of keywords that you may not want to include within a Moz Analytics campaign. In today’s Daily SEO Fix, James dives into the tool to show you how to use Rank Tracker to start keeping tabs on the ranking history of a given keyword. This video is part of The Moz Daily SEO Fix tutorial series--Moz tool tips and tricks in under 2 minutes. To watch all of our videos so far, and to subscribe to future ones, make sure to visit the Daily SEO Fix channel on YouTube.
Moz Bar | | kellyjcoop1 -
Domain.com isn't recognized by on-page-grader, but domain.com/index.php is
I am running a website through On-page-grader, as www.domain.com and scores an "F" for a specific keyword. When it's ran as www.domain.com/index.php, it scores an "A" for that same keyword and has everything checked other than "keyword in the domain name". There are no other files such as index.htm, or index.html that would interfere and can't figure out why this page is not being recognized. I checked, the robots and .htaccess file, but do not see anything that would hinder. Could this be a server issue?
Moz Bar | | werkbot0 -
Should I be getting an 'A' Grade for basic words in a page title?
It seems as long as a word in the page title matches a word(s) within the page content you will get an 'A' grade. Should I be replacing these keywords with higher monthly searches and lower difficulty? It appears deceptive when you test a new website and all the pages get an 'A' for a word as basic as 'about.' Please advise.
Moz Bar | | Joseph.Lusso0 -
Moz ranking report shows massive loss in Bing SERP's. But it is not the case?
My weekly ranking report shows massive loss of SERP's on Bing. For most keywords I was no longer in the top 50 which where first top 5. But when I checked it out for myself, I'm still ranked top 5 for these keywords. I then used the Moz Rank Tracker and there it also says I'm still ranking top 5 for these keywords. What should I believe now?
Moz Bar | | wellnesswooz0 -
Indexed pages
hello all, what does moz consider to be an "indexed page"? is it a number of words? paragraphs? thanks for the insight!
Moz Bar | | tm46150