605 : Page banned by robots.txt
-
Hello everyone,
I need experts help here, Please suggest, I am receiving crawl errors for my site that is , X-Robots-Tag: header, or tag.
my robots.txt file is:
User-agent: *
Disallow:
-
Hey there! I just followed up on the message you sent into our help team, but I wanted to also post the answer here for reference.
It looks like the robots.txt file may have recently been changed for the site because I created a new campaign for the subdomain and I am not getting that same error. You should no longer see this error on your next campaign update or you could create a new campaign and you would no longer see the error there.
I did notice that you ran a number of crawl tests on the site since the campaign update, but the important thing to realize is that the crawl test can be cached for up to 48 hours. (I removed the crawls in this version of the screenshot for privacy.) We also cache the crawl tests from campaign crawls, so it looks like the first crawl test you ran on the 29th was cached from your campaign crawl and the two subsequent crawl tests were cached from that first crawl test.
Again, I wanted to note that it looks like there are only links to about 2 other pages (terms and privacy) that are on the specific subdomain you are tracking, so we aren't able to crawl beyond those pages. When you limit a campaign to a specific subdomain, we can only access and crawl links that are within the same subdomain.
-
I am at a lost, I can't find the issue. Let us know what Moz says.
-
I actually have come across a handful URLs that are NoIndex, I'll DM you a list once complete.
I can't be certain this is the root of the problem (I've never seen this error in the crawl report), but based on the error you said you're getting, I believe it's a great starting point.
-
Hi Logan Ray
thank you for detailed guide, all tools bot are working perfectly except moz's. My robots meta is index, follow and my robots.txt is disallow for none for all user agents. Still there is confusion that why moz is showing crawl error. I have now emailed to moz. Let's see what they reply. I will share that.
thank you
-
Hi,
This sounds like it's more related to the meta robots tag, not the robots.txt file.
Try this:
- Run a Screaming Frog crawl on your site
- Once complete, go to the Directives tab
- Look for 'NoIndex' in the 'Meta Robots 1' column (should be the 3rd column)
- If you see anything marked with that tag, remove them - unless of course you need them there for a reason, in which case you should also block that page in your robots.txt file
-
Are you able to provide a link to site (DM me if you don't want it posted on the forum)
-
I am receiving crawl error for moz only.
There is no error at google's search console. Also, I have tested at google's robots.txt testing tool. https://www.google.com/webmasters/tools/robots-testing-too
My robots.txt file is with no slash.
User-agent: *
Disallow: -
Hi Bhomes,
Try clearing you robots.txt of any content, a robots.txt with:
User-agent: *
Disallow:/
Is blocking everything from crawling your site. See: https://support.google.com/webmasters/answer/6062598?hl=en for testing and more details on robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz bot has trouble crawling Angular JS - I believe it's seeing the SPA (Single Page Application) before Universal. Anyone else have this issue? What is the fix?
The Moz bot user agent detection settings are able to read Universal, but the Single Page Application (SPA) version partially loads on the website before Universal. Because of this, Moz (and possibly search engines) think we have massive duplicate content issues. For example, the crawl report said a particular product page (which has about 1,000 words) has 33,000 words and has duplicate content with over 300 other pages. This makes me believe it's only picking up the SPA version. Has anyone come across this, and what would be the fix?
Moz Bar | | laurengdicenso1 -
Looking for a Tool to Find Referring Pages of Specific URLs
Hello Everyone, We are Looking for a Tool to Find Referring Pages of Specific URLs. Please let me know if you know of a Moz tool or another tool for this need. Thanks.
Moz Bar | | Pushm4 -
How to find the pages in the site which is ranking for zero keyword
I have around three thousand pages in my website,how to find the list of pages which is ranking for zero keywords
Moz Bar | | srinivasan.n1 -
Moz Crawl only crawling the top level page (1 page)
For the past few mounts my weekly site crawl has been inconsistent. One week works fine, it crawls all of my 500 or so pages. The following week it only crawls 1 page (http://mydomain.com) and nothing else. A few weekly scan go by and the crawl is back up the the 500 or so pages.I went ahead and created several campaigns with duplicate settings and crawled the site. Most times but not all the new campaign's crawl works fine crawling all pages. But within a week or two the weekly crawl will fail again. (crawling 1 page). Currently i have four campaign's all with the same settings running weekly crawls. 2 campaign's crawled the 500 pages and two crawled only the single page. Any help will be greatly appreciated
Moz Bar | | dmaude0 -
Site Crawl report show strange duplicate pages
Beginning in early in Feb, we got a big bump in duplicate pages. The URLs of the pages are very odd: Example URL:
Moz Bar | | Neo4j
http://firstname.lastname@website.com/dir/page.php
is duplicate with http://website.com/dir/page.php I checked though the site, nginx conf files, and referral pages, and could not find what is prefixing the pages with 'http://firstname.lastname@'. Any ideas? The person whose name is 'Firstname Lastname' is stumped as well. Thanks.0 -
On-Page Grader "Sorry, but that URL is inaccessible."
We have a new client with a squarespace page. http://www.mountainhouseestate.com The Moz On-Page grader returns the error "Sorry, but that URL is inaccessible." for all pages. Possibly related, Google seems to hate their site. Even a search for "mountain house estate" returns lousy results. Bing/Yahoo has no problem with it.
Moz Bar | | Duke_Ferris1 -
Puzzled. High domain authority, but low page authority?
I have a question regarding how a page's authority can be low but the domain authority be high. I'm using the SERP report in moz to compare competing pages. The page I'm working on is an ecommerce category page. Moz says it has very low page authority, has been indexed for a little over a year, it's domain authority is a moderate 50, and has no root domains linking to it. It is more relevant than the other pages (has more products for sale that fit the keyword searched for), has a decent amount of copy on the page, but ranks around #10 in the search results. The competing pages are also ecommerce category pages. The 9 ranking higher have domain authority varying from 30-75, and 1 or 2 root domain links to them. Their number of products that match the keyword searched are all over the board, some have many matches, some have only 6 or 8 results. Some have a moderate amount of text on the page, some have very little text on the page. Yet they have page authority in the 30s-40s. What goes into page authority that I'm missing which would make these other pages have so much more authority when they are just slightly better in the other metrics listed above? What am I missing here?
Moz Bar | | K-WINTER0 -
Www.site.com linking to pages www10.site.com
The root domain of the website in question is www.site.com but all subpages are on the subdomain www10.site.com (I'm pretty sure it's a subdomain, at least, used for load balancing?). A funny thing happens on this site with the moz toolbar. I visit a subpage, www10.site.com/articles/articletopic1 That page has a lot of links on it, all of them visibly going to the subdomain www10.site.com. However, the moz toolbar shows some of them as Internal links and most of them as External links. As far as I can tell, there is no real rhyme or reason to the difference between the links that are highlighted as Internal vs. External. The link structures vary greatly: Some are properly structured www10.site.com/blogs/category
Moz Bar | | Motava
And some are poor like www10.site.com/articles/show_articles.php?section=category1 So a couple questions here: Does this subdomain www10 have a detriment on the rankings of subpages?
What could possibly cause the internal links on these subpages to be highlighted as external pages with the moz toolbar?1