Google Unable to Access Robots.txt
-
We haven't made any changes to the robots.txt file and suddenly Google claims they can no longer access the file.
The site has been up and active for well over a year now. What are my next steps?
I have included a screenshot of the top half of the file. See anything wrong?
-
Would you mind sharing the solution - if not related to domain intricacies
Curious to know - only for knowledge purpose
Many thanks
-
Thanks Phillip. I think we're on the way to discovering the issue.
I certainly appreciate the feedback!
-
I think you should allow general access before beginning to disallow various sections below:
Allow: /
Though semantically, I think your robots.txt is in order. But if Google says they cannot acces robots.txt it seems like they cannot physically access the file itself on yourdomain.com/robots.txt, not that they cannot crawl the page due to your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Number or percentage of new visitors impact Google rankings?
Hi all, Does the number/percentage of new visitors (from different IPs and countries) impact the Google rankings? If there are more number of new visitors, will Google favours the website in rankings considering the fact that new visitors are better than returning/same visitors? Thanks
Algorithm Updates | | vtmoz0 -
Google update January 2015
Hello, In January 2015, google changed its European Algorithm. The change decreased the ranking of some of our keywords but not all. See article for more evidence in google changing its algorithm. https://www.seroundtable.com/google-update-maybe-19760.html The biggest change was the keyword phrase ‘Wholesale Silver Jewellery’ which we ranked 1 in SERP, but now we’re nowhere to be seen. However, the change didn’t affect our keyword phrase ‘Wholesale Jewellery Silver’, ’Wholesale Silver’ and ‘Wholesale Jewellery. We’ve been through our data and see that all of our ’Silver Jewellery’ keyword phrases are no longer showing in the SERP. Further research has shown that our competitors were also dropped down the rankings for the same keyword phrase. Our question is: Why has this update affected certain keyword phrases, such as ‘silver jewellery’ but not ‘jewellery silver’ and how should we over come this? Additional Information
Algorithm Updates | | SilverStar1
If you type in our company name ‘Mainly Silver’ or ‘mainlysilver’ were still showing in SERP, however if you type ‘mainlysilver jewellery’ we’re no where to be found. We’ve even checked ‘site:mainlysilver.co.uk silver jewellery’ in google search and it returns with ‘no results found’. If you switch the keyword phrase, all our web pages are showing up Our website is - www.mainlysilver.co.uk0 -
How can I check Googles Page Cache ?
Hi I use to have a handy tool in Firefox (Google Toolbar) that was very handy for checking page ranks and what date a page had been cached. For a while with the newer versions of Firefox I cannot seem to locate this useful tool, Can anybody recommend any useful tools for checking the above. Thanks Adam
Algorithm Updates | | AMG1000 -
Why am I getting different Google SERP result for same keywords?
Hi Mozzers, I have noticed recently that Google (.com.au) has been serving up different SERP results for the same keywords. For example, one of our main keywords is "Car Loan". One result will show our site as ranking #5 organically from 242,000,000 results. A refresh of this search will then result in our site not ranking at all from 133,000,000 results. We have been noticing this happen only in the last few days & more frustrating is that Google is throwing up the SERP from 133,000,000 results more frequently. Would anyone know why this is occurring? And what can we do, if anything, to ensure we are shown regardless of how many results Google calls from? Is it from recent algo update & will it settle down over time? Any help would be greatly appreciated. (Just to add - I'm not gogged in to Google when completing this test & regularly clear cookies etc so I don't believe its a personalised search issue)
Algorithm Updates | | 360Finance0 -
Google Places Rank Replacement
Hi Guys We have recentley done work for a client where they ended up position 1 for their chosen keyword, which was great. Since then there rank has changed and they are now first in google places rank but not on organic search at all, where as before they were on both? Any suggestions on why this may have happened? Not to sure at why google would have replaced the organic rank with the places rank? Cheers Chris
Algorithm Updates | | MiracleCreative0 -
Source for how many searches are done on Google per day?
Hi, All! The figure I've seen going around is 3 billion and is attributed to ComScore, but in the comScore press release that was linked to (actually the one from Rand's article - http://www.seomoz.org/blog/21-tactics-to-increase-blog-traffic-2012) all I could find was percentage of market share, but no total numbers of searches. Anyone have a source on that? Thanks, Aviva
Algorithm Updates | | debi_zyx0 -
What is the critical size to reach for a content farm to be under google spot?
We're looking for building a content farm, as an igniter for another site, so there will be some duplicate content. Is it a good or a bad strategy in terms of SEO.
Algorithm Updates | | sarenausa0 -
Is there a way to know what rank my site is listed on google ?
My current client web page was listed at the 4th page 1 month ago. Im trying real hard to make him understand that the traffic from beiing on the first page is important and that he need to give me additionnal ressource to make it happen ( i don't prog at all). So i had the idea of checking every page to see whats is current rank. but instead of looking from page 1 to page X, i was wondering if there was something somewhere that could give me my rank right away. It woud help saving time. Thx.
Algorithm Updates | | Promoteam0