Search engine blocked by robots-crawl error by moz & GWT
-
Hello Everyone,.
For My Site I am Getting Error Code 605: Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag, Also google Webmaster Also not able to fetch my site, tajsigma.com is my site
Any expert Can Help please,
Thanx
-
When was your last crawl date in Google Webmaster Tools/Search Console? It may be that your site was crawled with some kind of problem with the robots.txt and hasn't been re-crawled since.
-
Yes , Exactly
I am also worried For that only, Can you please help to identify my site problem
Thnx
-
That's very strange. The robots.txt looks fine, but here's what I see when I search for your site on Google.
-
Headers look fine and as you correctly said your robots and meta robots are also ok.
I have also noted that doing a site:www etc in google search is also returning pages for your site so again showing it is being crawled and indexed.
To all intents and purposes it looks ok to me. Someone else may be able to shed more light on the issue if they have experienced this error to this degree.
-
www.tajsigma.com This the Domain For Query robots 605- code
-
Just to check it that the live one, or just a test in GSC. Can you send a link to your site maybe in a PM.
-
Yes, It is Also okay there see Attached screenshot
-
Have you followed the following and in Google Search Console tried testing your robots file.
If you are allowing all, I would maybe suggest simply removing your robots.txt all together so it defaults to just crawling everything.
-
Thanx, Tim Holmes For Your Quick reply
But My robots.txt File is
User-agent: *
allow: /Also in All pages i have Add Meta-tag
Then After Page is Not Getting Fetched or Crawl by GWT.
Thnx
-
Hello Falguni,
I believe the error is saying pretty much everything you need to know. Your Robots file or robots meta would appear to be blocking your site from being crawled.
Have you checked your robots.txt file in your root - or type in http://www.yourdomain.com/robots.txt
To ensure your site is being crawled and for robots to have complete access the following should be in place
**User-agent: ***
Disallow: To exclude all robots from the entire server**User-agent: ***
Disallow: **/**If it is the a meta tag causing the issue you will require, or have it removed to default to the below.As opposed to the following combinations which could result in some areas not being indexed, crawled etc
_Hope that helps
Tim_
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does anyone know how to fix this structured data error on search console? Invalid value in field "itemtype"
I'm getting the same structured data error on search console form most of my websites, Invalid value in field "itemtype" I take off all the structured data but still having this problem, according to Search console is a syntax problem but I can't find what is causing this. Any guess, suggestion or solution for this?
Intermediate & Advanced SEO | | Alexanders0 -
Dates on Google Search Results
Hello, I manage htts://globalrose.com When I search on Google for "Yellow Roses", "Yellow Roses Globalrose", or any search that might bring up one of our pages, sometimes our search results appear with dates right before the description. Does anyone know what this mean? Why they appear on some and not other pages? Here is a search result for example: Example Google Search Can someone please help clarify this for us?
Intermediate & Advanced SEO | | globalrose.com0 -
When "pruning" old content, is it normal to see an drop in Domain Authority on Moz crawl report?
After reading several posts about the benefits of pruning old, irrelevant content, I went through a content audit exercise to kick off the year. The biggest category of changes so far has been to noindex + remove from sitemap a number of blog posts from 2015/2016 (which were very time-specific, i.e. software release details). I assigned many of the old posts a new canonical URL pointing to the parent category. I realize it'd be ideal to point to a more relevant/current blog post, but could this be where I've gone wrong? Another big change was to hide the old posts from the archive pages on the blog. Any advice/experience from anyone doing something similar much appreciated! Would be good to be reassured I'm on the right track and a slight drop is nothing to worry about. 🙂 If anyone is interested in having a look: https://vivaldi.com https://vivaldi.com/blog/snapshots [this is the category where changes have been made, primarily] https://vivaldi.com/blog/snapshots/keyboard-shortcut-editing/ [example of a pruned post]
Intermediate & Advanced SEO | | jonmc1 -
Search engine keyword rank - easiest way to check the keywords that rank across website
What is the best keyword ranking tool you have used? I have used various tools by which I am expected to identify and input the keyword I want to track... However, recently I was introduced to Searchmetrics, which I think automatically pulls in the keywords a website ranks for, without the need for manual input from the SEO (I haven't used this tool yet, so apologies if I am incorrect!). Do any other rank trackers work like this? Thanks, Luke
Intermediate & Advanced SEO | | McTaggart1 -
External Redirects & SEO
This company page redirects their external clients links: https://www.coinbase.com/clients QUESTION: What effect does this type of redirection have on the SEO going to these client pages, for their clients Websites?
Intermediate & Advanced SEO | | mstpeter0 -
Duplicate Page Content Issues Reported in Moz Crawl Report
Hi all, We have a lot of 'Duplicate Page Content' issues being reported on the Moz Crawl Report and I am trying to 'get to the bottom' of why they are deemed as errors... This page; http://www.bolsovercruiseclub.com/about-us/job-opportunities/ has (admittedly) very little content and is duplicated with; http://www.bolsovercruiseclub.com/cruise-deals/cruise-line-deals/explorer-of-the-seas-2015/ This page is basically an image and has just a couple of lines of static content. Also duplicated with; http://www.bolsovercruiseclub.com/cruise-lines/costa-cruises/costa-voyager/ This page relates to a single cruise ship and again has minimal content... Also duplicated with; http://www.bolsovercruiseclub.com/faq/packing/ This is an FAQ page again with only a few lines of content... Also duplicated with; http://www.bolsovercruiseclub.com/cruise-deals/cruise-line-deals/exclusive-canada-&-alaska-cruisetour/ Another page that just features an image and NO content... Also duplicated with; http://www.bolsovercruiseclub.com/cruise-deals/cruise-line-deals/free-upgrades-on-cunard-2014-&-2015/?page_number=6 A cruise deals page that has a little bit of static content and a lot of dynamic content (which I suspect isn't crawled) So my question is, is the duplicate content issued caused by the fact that each page has 'thin' or no content? If that is the case then I assume the simple fix is to increase add \ increase the content? I realise that I may have answered my own question but my brain is 'pickled' at the moment and so I guess I am just seeking assurances! 🙂 Thanks Andy
Intermediate & Advanced SEO | | TomKing0 -
Places Listing in Search Results
Hi everyone, We have a company that hired us to set-up their Google Places listing for their 2nd location. The listing for the 1st location is very strong. Lots of reviews, Zagat rating, Knowledge graph, etc. In the search results the Google Places listing for the 1st location has merged with the website listing. You can see the link to the main site w/ a small grey google places listing directly below it. The client would like BOTH Google places listing to show up in the search results. They both show up on the map listing but not in the search results. Each location has its own listing in Google Places. We have also created different pages on the website for each location. Is there a way to get the search results to display places listings? I have noticed a few other business have done it by naming each of their multiple locations something slightly different. Then the search results seem to realize there are multiple locations and display the places listing in the search results. Anyone run into this? Any ideas? Thanks!!
Intermediate & Advanced SEO | | SeattleJoe0 -
Wordpress error
On our Google Webmaster Tools I'm getting a Severe Health Warning regarding our Robot.txt file reading: User-agent: *
Intermediate & Advanced SEO | | NileCruises
Crawl-delay: 20 User-agent: 008
Disallow: / I'm wondering how I can fix this and stop it happening again. The site was hacked about 4 months ago but I thought we'd managed to clear things up. Colin0