605 : Page banned by robots.txt
-
Hello everyone,
I need experts help here, Please suggest, I am receiving crawl errors for my site that is , X-Robots-Tag: header, or tag.
my robots.txt file is:
User-agent: *
Disallow:
-
Hey there! I just followed up on the message you sent into our help team, but I wanted to also post the answer here for reference.
It looks like the robots.txt file may have recently been changed for the site because I created a new campaign for the subdomain and I am not getting that same error. You should no longer see this error on your next campaign update or you could create a new campaign and you would no longer see the error there.
I did notice that you ran a number of crawl tests on the site since the campaign update, but the important thing to realize is that the crawl test can be cached for up to 48 hours. (I removed the crawls in this version of the screenshot for privacy.) We also cache the crawl tests from campaign crawls, so it looks like the first crawl test you ran on the 29th was cached from your campaign crawl and the two subsequent crawl tests were cached from that first crawl test.
Again, I wanted to note that it looks like there are only links to about 2 other pages (terms and privacy) that are on the specific subdomain you are tracking, so we aren't able to crawl beyond those pages. When you limit a campaign to a specific subdomain, we can only access and crawl links that are within the same subdomain.
-
I am at a lost, I can't find the issue. Let us know what Moz says.
-
I actually have come across a handful URLs that are NoIndex, I'll DM you a list once complete.
I can't be certain this is the root of the problem (I've never seen this error in the crawl report), but based on the error you said you're getting, I believe it's a great starting point.
-
Hi Logan Ray
thank you for detailed guide, all tools bot are working perfectly except moz's. My robots meta is index, follow and my robots.txt is disallow for none for all user agents. Still there is confusion that why moz is showing crawl error. I have now emailed to moz. Let's see what they reply. I will share that.
thank you
-
Hi,
This sounds like it's more related to the meta robots tag, not the robots.txt file.
Try this:
- Run a Screaming Frog crawl on your site
- Once complete, go to the Directives tab
- Look for 'NoIndex' in the 'Meta Robots 1' column (should be the 3rd column)
- If you see anything marked with that tag, remove them - unless of course you need them there for a reason, in which case you should also block that page in your robots.txt file
-
Are you able to provide a link to site (DM me if you don't want it posted on the forum)
-
I am receiving crawl error for moz only.
There is no error at google's search console. Also, I have tested at google's robots.txt testing tool. https://www.google.com/webmasters/tools/robots-testing-too
My robots.txt file is with no slash.
User-agent: *
Disallow: -
Hi Bhomes,
Try clearing you robots.txt of any content, a robots.txt with:
User-agent: *
Disallow:/
Is blocking everything from crawling your site. See: https://support.google.com/webmasters/answer/6062598?hl=en for testing and more details on robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
MozBar: Page Title (and other data) inconsistent with HTML
In using the MozBar Chrome extension, I have noticed that the data about a page that MozBar displays is sometimes inconsistent with the "head" tag in the HTML of the page. Am I misunderstanding where MozBar is getting the data it is displaying? Other SEO extensions like "META SEO inspector"do not display these inconsistencies. EupH5sG
Moz Bar | | clayjones0 -
In MozBar Page Analysis under General Attributes Country is empty or wrong
Some of our clients sites have a top level domain for Mexico, but in the Mozbar they doesn't show any country, is this related to a known bug? https://www.edacom.mx/ https://www.dropbox.com/s/a8jgr7ztk1im15k/Captura de pantalla 2019-08-15 12.41.10.png?dl=0 https://www.cliento.mx/ https://www.dropbox.com/s/a8jgr7ztk1im15k/Captura de pantalla 2019-08-15 12.41.10.png?dl=0 In the search console: "International Targeting report" the target is Ok to Mexico.
Moz Bar | | Andrea_Ugalde1 -
Top pages according Link Explorer shows 301 pages and images
I was looking at the top pages on my website using Link Explorer, it contains 301 pages and image URLs. Is this expected or I should be doing something to fix it? URL with second highest PA in site is a page already 301 redirected more than 6 months back. URL with highest PA in my blog is a random image in the blog, rather than any blog posts. Is this normal? Thanks -Aji
Moz Bar | | ajiabs1 -
On Page Grader - URL not accessible
We have tried to use the On Page Grader today and it is coming back with URL not accessible for all pages on our website. We previously used the On Page Grader on Friday 10th Nov for a couple of product pages with no issues. Since then, the only changes we have made on the websites is updating some downloadable documents. We have done this several times before and it has never affected Moz. We have not changed the page URLs, and therefore do not know why it is now not working. The pages are working fine on the website with no issues. A link to one of the pages is below. http://www.processinstruments.co.uk/products/dissolved-oxygen-monitor/ Any help would be greatly appreciated.
Moz Bar | | PiMike0 -
Comparing Competitor On-Page Grades?
When I run an advanced SERP report or even the basic keyword difficultly report it gives me competitor On-page grades. I can look directly at their code, but I'm wondering if there is a way to easily see and export a .csv or excel file of the on-page ranking factors for the top 1-10 results comparing them to a page on my site? Seems like that would be really helpful and much quicker than looking at source code to understand what on-page factors are contributing to their success. Is there any way to do this in Moz or a free tool elsewhere you would suggest?
Moz Bar | | K-WINTER0 -
Should I be getting an 'A' Grade for basic words in a page title?
It seems as long as a word in the page title matches a word(s) within the page content you will get an 'A' grade. Should I be replacing these keywords with higher monthly searches and lower difficulty? It appears deceptive when you test a new website and all the pages get an 'A' for a word as basic as 'about.' Please advise.
Moz Bar | | Joseph.Lusso0 -
Duplicate page content/page titles on tages
Hello everyone. New to the community and loving it already. Question, I am receiving an error of 6 pages with duplicate content and page titles. A majority of these are tag pages. Should I be worried about these? IN the column listed duplicate urls it is listing 0 ( screen shot: http://screencast.com/t/azvuVk0ucWt) Are these tags a problem? Will SEO be hurt because of this? What are TAG pages? Actually pages, categories, should I eliminate these?
Moz Bar | | Jasonalanmagic0 -
Rank Tracker page shows only first 50 entries
On the Rank Tracker tool page, http://moz.com/researchtools/rank-tracker, when I hit the Next 25 it works the first time but none of the others. Trying to run a report and since we have more than 50 keywords I have to manually click on them to be checked.
Moz Bar | | HopeIndu0