605 : Page banned by robots.txt
-
Hello everyone,
I need experts help here, Please suggest, I am receiving crawl errors for my site that is , X-Robots-Tag: header, or tag.
my robots.txt file is:
User-agent: *
Disallow:
-
Hey there! I just followed up on the message you sent into our help team, but I wanted to also post the answer here for reference.
It looks like the robots.txt file may have recently been changed for the site because I created a new campaign for the subdomain and I am not getting that same error. You should no longer see this error on your next campaign update or you could create a new campaign and you would no longer see the error there.
I did notice that you ran a number of crawl tests on the site since the campaign update, but the important thing to realize is that the crawl test can be cached for up to 48 hours. (I removed the crawls in this version of the screenshot for privacy.) We also cache the crawl tests from campaign crawls, so it looks like the first crawl test you ran on the 29th was cached from your campaign crawl and the two subsequent crawl tests were cached from that first crawl test.
Again, I wanted to note that it looks like there are only links to about 2 other pages (terms and privacy) that are on the specific subdomain you are tracking, so we aren't able to crawl beyond those pages. When you limit a campaign to a specific subdomain, we can only access and crawl links that are within the same subdomain.
-
I am at a lost, I can't find the issue. Let us know what Moz says.
-
I actually have come across a handful URLs that are NoIndex, I'll DM you a list once complete.
I can't be certain this is the root of the problem (I've never seen this error in the crawl report), but based on the error you said you're getting, I believe it's a great starting point.
-
Hi Logan Ray
thank you for detailed guide, all tools bot are working perfectly except moz's. My robots meta is index, follow and my robots.txt is disallow for none for all user agents. Still there is confusion that why moz is showing crawl error. I have now emailed to moz. Let's see what they reply. I will share that.
thank you
-
Hi,
This sounds like it's more related to the meta robots tag, not the robots.txt file.
Try this:
- Run a Screaming Frog crawl on your site
- Once complete, go to the Directives tab
- Look for 'NoIndex' in the 'Meta Robots 1' column (should be the 3rd column)
- If you see anything marked with that tag, remove them - unless of course you need them there for a reason, in which case you should also block that page in your robots.txt file
-
Are you able to provide a link to site (DM me if you don't want it posted on the forum)
-
I am receiving crawl error for moz only.
There is no error at google's search console. Also, I have tested at google's robots.txt testing tool. https://www.google.com/webmasters/tools/robots-testing-too
My robots.txt file is with no slash.
User-agent: *
Disallow: -
Hi Bhomes,
Try clearing you robots.txt of any content, a robots.txt with:
User-agent: *
Disallow:/
Is blocking everything from crawling your site. See: https://support.google.com/webmasters/answer/6062598?hl=en for testing and more details on robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
On-Page Grader giving wrong results for missing keyword in image Alt Attribute
I'm trying to optimize for the Keyword "River City Apartments" on this page: https://3fconstruction.net/case-studies/river-city-apartments-chicago-condo-deconversion-renovation/ When I use On-Page Grader to see how I'm doing it's telling me I'm missing the Keyword in the Image Alt Attribute. However, it's clearly in there. <strong>River City Apartments Aerial</strong> What am I missing?
Moz Bar | | Drew.Friestedt0 -
Moz Crawler not Identifying all Duplicate Pages
On two recent site crawls (9/27/14 and 11/4/14) for duplicate content the Moz tool did not ID the following 2 pages, which are 100% duplicate to each other: http://www.hooksandlattice.com/planter-hampton-241212.html ; Screenshot: http://screencast.com/t/DdwWroUU http://www.hooksandlattice.com/planter-hampton-721212.html ; Screenshot: http://screencast.com/t/8Lb1cJZmGrhX As I'm working feverishly to re-write and update the site (goal is ZERO duplicates) I'm finding it challenging to use the Moz tool to get the project done. Does anyone have any feedback or help they can provide for how I can identify all duplicate pages associated with my domain? Thank you! Lindsey Pfeiffer
Moz Bar | | CMC-SD0 -
Moz Local | Empty page "Categories"
Dear Moz, Another error, the following url loads an empty page https://moz.com/local/categories Please review Thanks!
Moz Bar | | Bio-RadAbs0 -
On the on page optimization page, I found out that there are 2 contributing factors which are opposite to each other. "No More Than One H1 Tag" and "Appropriate Keyword Usage in H1 Tag"
"No More Than One H1 Tag" and "Appropriate Keyword Usage in H1 Tag" If you fulfill one condition, the other one is not completed. If you consider Article heading as H1 then Moz do not detect keyword in the heading.
Moz Bar | | MoeezLodhi0 -
On-Page grader
Hi Have you changed criteria for onpage grader recently since i see a page a havn't touched/changed has dropped from an A to B ? Cheers Dan
Moz Bar | | Dan-Lawrence0 -
Duplicate content - Which is the other duplicate page?
Hi I just ran a campaign, and I got a duplicate content warning for some of my pages. When I go into the diagnostic report, I am unable to find the page detected by google as 'duplicate' to the main page. Unless I know which 2 pages are being detected as duplicate, it'll be really difficult to actually solve the problem. Would be great to have any kind of help here. Thanks in advance!
Moz Bar | | rjchugh0 -
How does an index page have a higher Authority than the root domain?
So just curious, but on my domain, http://www.bulwarkpestcontrol.com the Page Authority is 59 and the root domain authority 52. That seems odd as it is the same page. Explanation?
Moz Bar | | Thos0030 -
Duplicate page content
Hi guys the feedback form my campaign suggests I have to much duplicate page content. I’ve had a look at the CSV file but it doesn’t seem to be abundantly clear as to which pages on my site have the duplicate content. Can anyone tell which columns I need to refer to on the sheet, to ascertain this information. Also if the content is only slightly different, will Google still consider it to be duplicate? I look forward to hearing from you
Moz Bar | | Hardley1110