605 : Page banned by robots.txt
-
Hello everyone,
I need experts help here, Please suggest, I am receiving crawl errors for my site that is , X-Robots-Tag: header, or tag.
my robots.txt file is:
User-agent: *
Disallow:
-
Hey there! I just followed up on the message you sent into our help team, but I wanted to also post the answer here for reference.
It looks like the robots.txt file may have recently been changed for the site because I created a new campaign for the subdomain and I am not getting that same error. You should no longer see this error on your next campaign update or you could create a new campaign and you would no longer see the error there.
I did notice that you ran a number of crawl tests on the site since the campaign update, but the important thing to realize is that the crawl test can be cached for up to 48 hours. (I removed the crawls in this version of the screenshot for privacy.) We also cache the crawl tests from campaign crawls, so it looks like the first crawl test you ran on the 29th was cached from your campaign crawl and the two subsequent crawl tests were cached from that first crawl test.
Again, I wanted to note that it looks like there are only links to about 2 other pages (terms and privacy) that are on the specific subdomain you are tracking, so we aren't able to crawl beyond those pages. When you limit a campaign to a specific subdomain, we can only access and crawl links that are within the same subdomain.
-
I am at a lost, I can't find the issue. Let us know what Moz says.
-
I actually have come across a handful URLs that are NoIndex, I'll DM you a list once complete.
I can't be certain this is the root of the problem (I've never seen this error in the crawl report), but based on the error you said you're getting, I believe it's a great starting point.
-
Hi Logan Ray
thank you for detailed guide, all tools bot are working perfectly except moz's. My robots meta is index, follow and my robots.txt is disallow for none for all user agents. Still there is confusion that why moz is showing crawl error. I have now emailed to moz. Let's see what they reply. I will share that.
thank you
-
Hi,
This sounds like it's more related to the meta robots tag, not the robots.txt file.
Try this:
- Run a Screaming Frog crawl on your site
- Once complete, go to the Directives tab
- Look for 'NoIndex' in the 'Meta Robots 1' column (should be the 3rd column)
- If you see anything marked with that tag, remove them - unless of course you need them there for a reason, in which case you should also block that page in your robots.txt file
-
Are you able to provide a link to site (DM me if you don't want it posted on the forum)
-
I am receiving crawl error for moz only.
There is no error at google's search console. Also, I have tested at google's robots.txt testing tool. https://www.google.com/webmasters/tools/robots-testing-too
My robots.txt file is with no slash.
User-agent: *
Disallow: -
Hi Bhomes,
Try clearing you robots.txt of any content, a robots.txt with:
User-agent: *
Disallow:/
Is blocking everything from crawling your site. See: https://support.google.com/webmasters/answer/6062598?hl=en for testing and more details on robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Page Title showing Twice when I check it through Moz Bar
Hello Experts, I have a brand new website. And I know I have to do lot of things and make changes for better SEO. With that in mind, I recently installed Mozbar. I was checking one of my posts the other day and noticed Page Title is appearing twice in Mozbar.Please see the screenshot below. Could you please guide me what could be causing this? FYI, I have installed Yoast SEO Plugin. Also, I can see "Alt Text" is picking lot many words which are from other images or posts...Is there any way to customize that as well? KhP9y
Moz Bar | | techierichard0 -
Is there anyway to track on-page grader grades for a site?
As opposed to going through the process of inputting the urls and keywords every couple of weeks, is there a way of Moz to send the scores for those keywords and the domain to which the url belongs?
Moz Bar | | DigitalGuru0 -
On-Page Grader supporting HTTPS?
Hi there, I am getting the error "Sorry, but this URL is inaccessible" on the Moz On-Page Grader. I have tested with multiple URL's but any that link to my site give the error. I have noticed my site's URL's redirect to https so my question is..is the on-page grader in-compatible with https? Cheers!
Moz Bar | | ToteIreland
Ross0 -
I'm getting a Crawl error 605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
The website is www.bigbluem.com and is a wordpress site. I'm getting the following error: 605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag But what is weird is the domain it lists below that is http://None/BigBlueM.com Any advice?
Moz Bar | | TumbleweedPDX1 -
On-Page Grader is not counting H2 and H3 Keywords
I used on-page grader in one of my webpages and it showed the limit of 15 keywords used. The problem is that it counted only when the keyword appears in the Title, URL, Meta Description, H1, Body and IMG ALT. If you take into account the H2 and H3 tags, the count will go to 22. Is there a problem with the On-Page Grader Tool or it's not a problem to go above 15 keywords if they are in H2 and H3? Here's the link for the page: http://www.tororadar.com.br/acoes/bbse3-bb-seguridade/cotacoes-bovespa Keyword: BBSE3 Thanks in advance.
Moz Bar | | Guilhermealvesc0 -
On Page Grader Changed?
Hi, Could anyone tell me if the on page grader has changed? I remember seeing 8 boxes at the top but now i'm only seeing 6... Before I remember seeing a H2-H4 and a Strong/B Box? Thanks
Moz Bar | | dmxsta0 -
On page grades - quick question
My hardwood flooring client just added carpet, luxury vinyl, ceramic tile, area rugs, design, and ceramic tile to his lineup - joy, joy. He really thought the months and months of working together and creating a bunch of solid, original content and promoting it and building links and seeing great results for Hardwood Flooring means he would instantly rank well for ALL flooring. I digress, sorry. I am trying to understand the on page reports a little better. When I grade the new carpet page on his site for the keyword 'carpet' it comes back with an A - when I grade it for 'Carpet, city name' it comes back with an F. Which should I be targeting for best results? 95% of my clients are concerned about local results and not getting business other than within 20-30 miles of their office. Any guidance in understanding on how BEST to utilize the on page reporting would be greatly appreciated! (I did watch the video and check out the guide) Thank you in advance for your time! Matthew
Moz Bar | | Mrupp440