Google Unable to Access Robots.txt
-
We haven't made any changes to the robots.txt file and suddenly Google claims they can no longer access the file.
The site has been up and active for well over a year now. What are my next steps?
I have included a screenshot of the top half of the file. See anything wrong?
-
Would you mind sharing the solution - if not related to domain intricacies
Curious to know - only for knowledge purpose
Many thanks
-
Thanks Phillip. I think we're on the way to discovering the issue.
I certainly appreciate the feedback!
-
I think you should allow general access before beginning to disallow various sections below:
Allow: /
Though semantically, I think your robots.txt is in order. But if Google says they cannot acces robots.txt it seems like they cannot physically access the file itself on yourdomain.com/robots.txt, not that they cannot crawl the page due to your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does Google’s Algorithm Populate Answer Boxes with Its Own Independent Research?
If you search 'best games to play for youtube' you get an answer box with answers pulled independently from the article at hand. Here's an image: http://imgur.com/a/S0j9B Here are all the games from my article, in the order in which they appear. Google's chosen games for Answer Box are bolded: Battlefield 1 Bloodborne GTA V FiFA 16 TrackMania Turbo Garry’s Mod League of Legends Call of Duty: Black Ops III Tom Clancy’s The Division Overwatch Just Cause 3 Counter-Strike: Global Offensive Brawlhalla Rocket League Dark Souls III Unravel Firewatch GoldenEye 007 (this was put in as a joke, but coded as an H2 nonetheless) Destiny Dead by Daylight Fallout 4 Undertale No Man’s Sky Minecraft As you can see, Google is choosing which games to display to its searchers. My Crazy Egg data shows that these were not picked by click volume (each of these H2s are hyperlinked), which means Google must be using some other popularity metric, such as its own search volume data or external sales data. I wrote this up in a post on my site, for anybody who's curious.
Algorithm Updates | | Edward_Sturm1 -
Any Google Algorithm update?
Did Google release any seo update in last couple of days? My client is seeing 33% low traffic. The site is hyper-local marketplace website in India.
Algorithm Updates | | Avin1230 -
Has Google Authorship been completely removed from SERPs?
Noticed today that when I search (non-personalised search, incognito etc.) some of my pages on Google ALL references to authorship have now been completely removed. Does anyone know when this change occurred? I might be a bit slow this week (or last week) with concentrating on projects. I know like others that photos went some time back but now there are no author details being displayed. Just the page title and description. David
Algorithm Updates | | David-E-Carey1 -
Why is Google.co.uk showing so many US retailers on search queries?
Over the last couple of years Google seems to be showing more and more irrelevant search results in the field I am based in (dietary supplements) with searches for products coming back with results which highlight US vendors. Given I am in google.co.uk and actually logged in as well, why is Google so far off with their results lately?
Algorithm Updates | | predatornutrition0 -
Google Places/Points of Interest Rankings?
Does anyone have an idea on how Google ranks or determines the 'Points of Interests' that come up when searching about places/cities?
Algorithm Updates | | CarlLarson0 -
How will SEO be impacted by Google's new Knowledge Graph?
With the recent announcement of Googles new Knowledge Graph, the SERP will be different. Will this present a new set of SEO best practices?
Algorithm Updates | | PerriCline0 -
Can Google display a diffrent page title?
Hi if I search google UK for the phrase car leasing, google returns my listing as Car Lease Deals However the same search on Yahoo or Bing bring back Contract Hire | Vehicle & Car Leasing Deals | Car Lease Deals this is the real page title. Why would this happen? Thanks Andy
Algorithm Updates | | First-VehicleLeasing0 -
Anyone have stats on numbers of Google users searching while logged in?
In light of Google's recent "social search update", I am curious to know how many Google users perform searches while logged into their Google account thereby showing "social results".
Algorithm Updates | | Gyi0