Google Unable to Access Robots.txt
-
We haven't made any changes to the robots.txt file and suddenly Google claims they can no longer access the file.
The site has been up and active for well over a year now. What are my next steps?
I have included a screenshot of the top half of the file. See anything wrong?
-
Would you mind sharing the solution - if not related to domain intricacies
Curious to know - only for knowledge purpose
Many thanks
-
Thanks Phillip. I think we're on the way to discovering the issue.
I certainly appreciate the feedback!
-
I think you should allow general access before beginning to disallow various sections below:
Allow: /
Though semantically, I think your robots.txt is in order. But if Google says they cannot acces robots.txt it seems like they cannot physically access the file itself on yourdomain.com/robots.txt, not that they cannot crawl the page due to your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Meta robots at every page rather than using robots.txt for blocking crawlers? How they'll get indexed if we block crawlers?
Hi all, The suggestion to use meta robots tag rather than robots.txt file is to make sure the pages do not get indexed if their hyperlinks are available anywhere on the internet. I don't understand how the pages will be indexed if the entire site is blocked? Even though there are page links are available, will Google really index those pages? One of our site got blocked from robots file but internal links are available on internet for years which are not been indexed. So technically robots.txt file is quite enough right? Please clarify and guide me if I'm wrong. Thanks
Algorithm Updates | | vtmoz0 -
Advice regarding latest Google Algo Update please, if possible...
Hi there I wonder if anyone can advise on this. Since the latest google update on 1st Sept or whenever it rolled out, we noticed an initial spike in hits on our site, which was great. However now we are noticing levels going back to where they were and less people visiting the site. It also seems to be very sporadic. So we have a period of say a couple of hours with no one on the site, then suddenly loads visiting. We have also noticed a big dip in enquiries, despite the site having roughly the same amount of visitors. All our stats on Moz, Webmaster Tools, Ahrefs, Serpfox and various other rank trackers are showing that we have had an increase in visibility on our tracked keywords. There is a definite spike on all, but where is our traffic and where are our enquiries? Usually we are able to work out where the problem is when updates occur but with this we have no idea. We are utterly baffled. Is this normal? Is this just fluctuations and will settle down? Has anyone else noticed weird things happening? If anyone has any ideas or experience of this then would be most grateful for any advice. Feeling rather desperate at the moment. Many thanks in advance. Clojo
Algorithm Updates | | Clojobobo0 -
Google keyword tool
I was quite happy with google keyword tool for basic and accurate searches for keywords. Can anyone suggests a new tool that will give accurate search volume on google ( country specific ) I am not interest in info for adwords, and find a keyword planner tool way out in traffic results, compared to Keyword tool. Is the keyword tool completely gone?
Algorithm Updates | | summer3000 -
How do I separate 2 Google+ business listings?
Ever since Google Places started merging with Google+, my client's business listing is now showing up in local search results incorrectly under another business name who shares the same address as them. Has anyone else encountered this problem or a way to correct it?
Algorithm Updates | | TheeDigital0 -
Google and Content at Top of Page Change?
We always hear about how Google made this change or that change this month to their algorithm. Sometimes it's true and other times it's just a rumor. So this week I was speaking with someone in the SEO field who said that this week a change occurred at Google and is going to become more prevalent where content placed at the "top of the fold" on merchant sites with products are going to get better placement, rather than if you have your products at top with some content beneath them at the bottom of the page. Any comments on this?
Algorithm Updates | | applesofgold0 -
Can AJAX implementation affect the rankings in Google Panda?
Hi there, I have the following situation with one of our job sites. We migrate the site to a new application, which is better from design point of view and also usability. For this we use a lot AJAX especially in searches. So every time a user is filtering down their search new results will be shown on the page, at the same url and with no page load. But, having this implementation. affected Bounce rate - which increased from 38% to nearly 60%, PI/visits - which are now half, at 3 and also Avg Time on Site is half that is used to be coming to 2,5 min from nearly 6 min. From Rand post, it is clearly that the content is very important in Google Panda, and all of these parameters we should consider, as it is telling the quality of the content. So, my question will be, can this site be hit by Panda updates (maybe later on) because Bounce Rate, PI/Visits and Avg Time on site, decreased in such way? At the moment we don't measure the Ajax impresion, but as I understood that we can do that though virtual pages in GA, does anyone of you have the experience how to handle this? Won't be this an artificial increase? Thanks, Irina
Algorithm Updates | | InformMedia0 -
Strange Refferral URL coming in from Google
Hi, I've been monitoring my referral URL's coming in and today noticed they had changed. Previously when I clicked one it would be the google search result page - however now they all seem to be like this: http://www.google.co.uk/url?sa=t&source=web&cd=7&sqi=2&ved=0CHEQFjAG&url=http://www.mysite.com&rct=j&q=my%20keyword&ei=Bvc3TrbgB5G0hAfvqoSvAg&usg=AFQjCNFONDCPJDl3d2PYceYvale_cL7s4Q All these URL's immediately redirect to my website pages. Do you know what they are - they seem to be tracking URL's of some sort I am thinking?? Are they trying to analyse my site with respect to certain keywords?? Thanks
Algorithm Updates | | James770 -
Is there a way to know what rank my site is listed on google ?
My current client web page was listed at the 4th page 1 month ago. Im trying real hard to make him understand that the traffic from beiing on the first page is important and that he need to give me additionnal ressource to make it happen ( i don't prog at all). So i had the idea of checking every page to see whats is current rank. but instead of looking from page 1 to page X, i was wondering if there was something somewhere that could give me my rank right away. It woud help saving time. Thx.
Algorithm Updates | | Promoteam0