Google Unable to Access Robots.txt
-
We haven't made any changes to the robots.txt file and suddenly Google claims they can no longer access the file.
The site has been up and active for well over a year now. What are my next steps?
I have included a screenshot of the top half of the file. See anything wrong?
-
Would you mind sharing the solution - if not related to domain intricacies
Curious to know - only for knowledge purpose
Many thanks
-
Thanks Phillip. I think we're on the way to discovering the issue.
I certainly appreciate the feedback!
-
I think you should allow general access before beginning to disallow various sections below:
Allow: /
Though semantically, I think your robots.txt is in order. But if Google says they cannot acces robots.txt it seems like they cannot physically access the file itself on yourdomain.com/robots.txt, not that they cannot crawl the page due to your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Traffic cut-off since Google core update
Hi all, I am the webmaster of www.chepicap.com/en (Cryptocurrency news), and since the 3rd of june (Google core algorithm update) we got the hammer from Google. Organic traffic dropped with 90%+ overnight. We are still in the dark whether we can do to improve the current situation. Does someone have suggestions regarding this issue?
Algorithm Updates | | NielsDE0 -
How Google distinguish and ignore keyword attested with or in a brand?
Hi community, Generally there will be a primary keyword which everybody concentrates and expect their homepage or website to rank for....like "seo" for seo consulting or seo tools. There might be some companies with this keyword in their brand name like "ABC SEO". So this primary keyword will be all over the website being part of the brand name; especially in page titles and header tags. How Google distinguish and ignores this keyword in brand name to avoid giving more ranking boost to such websites? Will this keyword will be completely ignored being the part of the company name or their domain name? How Google distinguish between a generic keyword and keyword in company name? Thanks
Algorithm Updates | | vtmoz0 -
Number or percentage of new visitors impact Google rankings?
Hi all, Does the number/percentage of new visitors (from different IPs and countries) impact the Google rankings? If there are more number of new visitors, will Google favours the website in rankings considering the fact that new visitors are better than returning/same visitors? Thanks
Algorithm Updates | | vtmoz0 -
When Google is going to penalise webpages with autoplaying videos?
Hi, One of the most disgusting experiences while browsing web pages is the auto playing anonymous and highly non relevant ad videos at some corner of the page which buffer mostly and automatically plays. There are familiar websites which been following this activity. I don't think Google has ever rolled out an update to penalise these. Any idea why? Thanks
Algorithm Updates | | vtmoz0 -
Google's Presentation Yesterday
We hired a new website/marketing company that is a Preferred Google Partner (one of two in Charlotte according to them) and they hosted a presentation by Google at the Google Fiber office in Charlotte yesterday. As expected, there were lots of self-promotion by Google, accompanied with a plethora of data they created to support their PPC Marketing. It was an impressive performance with Molly Dince and Celena Fergusson, presenting Google Marketing Solutions: "Making the Web Work For You" and the keynote speaker Tim Reis, Director of Performance Agencies at Google: speaking on "Mobile Micromoments: Why Your Biggest Opportunities Are In The Smallest Moments" They ended with 15 minutes of Q&A and my question was answered with "I don't know" which I found surprising. So, here it is Thursday morning and I'm asking the same question to my Moz Family for some feedback: "Since the removal of Ads from the right column of a SERP, what percentage of Google traffic comes from Ads vs. the Organics?" I look forward to your comments. TY,
Algorithm Updates | | KevnJr
KJr0 -
How to fix Yahoo/Bing Ranking with hurting great Google ranking
If you have a Top ranking for keyword in Google but for Bing and Yahoo you rank considerably lower how do you balance the desire to rank better in Yahoo/Bing with not wanting to damage your Google ranking? Have people found certain on page SEO tactics help one but damage the other? Does anyone else have great Google rankngs for keywords but Bing/Yahoo are mediocre to poor?
Algorithm Updates | | inhouseninja0 -
Is Google now ignoring title tags?
Hey guys, I noticed alot of titles of webpages in Google now vary from search to search. They also differ from the Title tag that has been set by the webmaster. Anyone else notice this? (My results are depersonalized)
Algorithm Updates | | benjaminspak0 -
How much does Google take Social into account in serps
I have been reading and trying to learn more about how google takes social media into account when ranking sites, but I cant seam to find a definitive answer to this, does it make a big difference or does it not really matter?
Algorithm Updates | | MiracleCreative0