Google Unable to Access Robots.txt
-
We haven't made any changes to the robots.txt file and suddenly Google claims they can no longer access the file.
The site has been up and active for well over a year now. What are my next steps?
I have included a screenshot of the top half of the file. See anything wrong?
-
Would you mind sharing the solution - if not related to domain intricacies
Curious to know - only for knowledge purpose
Many thanks
-
Thanks Phillip. I think we're on the way to discovering the issue.
I certainly appreciate the feedback!
-
I think you should allow general access before beginning to disallow various sections below:
Allow: /
Though semantically, I think your robots.txt is in order. But if Google says they cannot acces robots.txt it seems like they cannot physically access the file itself on yourdomain.com/robots.txt, not that they cannot crawl the page due to your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Anchor name URLs & anchor blocks: how Google sees them?
Hi guys, Anchor name URLs & anchor blocks: how Google sees them? As far as I know Google hasn't ever recommended anchor name URLs and anchor blocks, mostly when you have one page site, but I have ran into an organic result with an hyper-link to an anchor name URL. anchor name link There is a proper link and there aren't on the page and the code the words "Jump to". It means Google has put those words there and it has also taken the header of that block as anchor text. Why has Google placed that link? The query is "faqs umbrella company", so I thought that Google has seen "faqs umbrella company" like "what is the most popular faq about umbrella companies?" and therefore perhaps the correct answer could be "Is an umbrella company the only option I have? What are the alternatives?". Although, IMHO the most popular FAQ on Umbrella Companies should always be "what is an umbrella company". Unfortunately, that page is only worthy of third Google organic result page and there is no hint of rich snippet or any kind of conversational/KBT optimisation on its source code. no-rich-snippet Someone has any idea of why Google shows that link and if it's something that we can optimise in our pages? Cheers Pierpaolo IhwGwkb.jpg VWORt5F.jpg
Algorithm Updates | | madcow780 -
Changes in Google "Site:" Search Algorithm Over Time?
I was wondering if anyone has noticed changes in how Google returns 'site:' searches over the past few years or months. I remember being able to do a search such as "site:example.com" and Google would return a list of webpages where the order may have shown the higher page rank pages (due to link building, etc) first and/or parent category pages higher up in the list of the first page (if relevant) first (as they could have higher PR naturally, anyways). It seems that these days I can hardly find quality / target pages that have higher page rank on the first page of Google's site: search results. Is this just me... or has Google perhaps purposely scrambled the SERPS somewhat for site: searches to not give away their page ranking secrets?
Algorithm Updates | | OrionGroup1 -
Profile Picture Display Next to Map on Google Search
On a recent Google search, I noticed that the top result also displayed the Google map of the location next to it but aside from that, it also displayed that business' Google+ Profile picture. I'm wondering how that is done. We have a Google+ account as well with an image associated with our business. At this point, the map doesn't even display for our business, but how does one get the map and the profile picture to appear on search results? Here's a link to the search result in question:
Algorithm Updates | | atuomala
http://www.google.com/search?q=tech+squad&rlz=1C1CHFX_enUS511US511&aq=f&oq=tech+squad&sourceid=chrome&ie=UTF-8#hl=en&rlz=1C1CHFX_enUS511US511&sclient=psy-ab&q=tech+squad+wi&oq=tech+squad+wi&gs_l=serp.3..0l2j0i30l2.6041.6590.0.6592.3.3.0.0.0.0.79.158.2.2.0.les%3B..0.0...1c.1.4.psy-ab.DoA-PSlyt-A&pbx=1&bav=on.2,or.r_gc.r_pw.r_qf.&bvm=bv.42768644,d.cGE&fp=b5d841bb1ab219dc&biw=1440&bih=785 tech_squad_serp.jpg0 -
How do I separate 2 Google+ business listings?
Ever since Google Places started merging with Google+, my client's business listing is now showing up in local search results incorrectly under another business name who shares the same address as them. Has anyone else encountered this problem or a way to correct it?
Algorithm Updates | | TheeDigital0 -
Client's site dropped completely from Google - AGAIN! Please help...
ok guys - hoping someone out there can help... (kinda long, but wanted to be sure all the details were out there) Already had this happen once - even posted in here about it - http://www.seomoz.org/q/client-s-site-dropped-completely-for-all-keywords-but-not-brand-name-not-manual-penalty-help Guy was a brand new client, all we did was tweak title tags and add a bit of content to his site since most was generic boilerplate text... started on our KW research and competitor research... in just a week, from title tag and content tweaks alone, he went from ranking on page 4-5 to ranking on page 3-4... then as we sat down to really optimize his site... POOF - he was gone from the Googs... He only showed up in "site:" searches and for exact matches of his business name - everything else was gone. Posted in here and on WMT - had several people check it out, both local guys and people from here (thanks to John Doherty for trying!) - but no one could figure out any reason why it would have happened. We submitted a reconsideration request, explaining that we knew we hadn't violated any quality guidelines, that he had less than 10 backlinks so it couldn't be bad linking, and that we had hardly touched the site. They sent back a canned response a week later that said there was no manual penalty and that we should "check our content" - mysteriously, the site started to show back up in the SERPs that morning (we got the canned response in the afternoon) There WAS an issue with NAP mismatch on some citations, but we fixed that, and that shouldn't have contributed to complete disappearance anyway. SO - the site was back, and back at its page 3 or 4 position... we decided to leave it alone for a few days just to be sure we didn't do anything... and then just 6 days later, when we were sitting down to fully optimize the site - POOF - completely gone again. We do SEO for a lot of different car dealers all over the country, and i know our strategies work. Looking at the competition in his market, he should easily be ranked page 2 or 3 with the very minimal tweaking we did... AND, since we didn't change anything since he came back, it makes even less sense that he was visible for a week and then gone again. So, mozzers... Anybody got any ideas? I'm really at a loss here - it makes zero sense that he's completely gone, except for his biz name... if nothing else, he should be ranking for "used cars canton"... Definitely appreciate any help anyone can offer -
Algorithm Updates | | Greg_Gifford0 -
Changing Googles Sitelinks
Hi all, I know Google will only show sitelinks if the site is deemed authoritive and if it will help the user searching a keyword, but is there anyway to order or control which links appear in the sitelinks? I know you can demote a sitelink in Webmasters, but is this not shooting yourself in the foot? If I demote a link will Google replace it with the next link it thinks is worthwhile and be doing this eventually show the links you want to appear in your sitelinks? Thanx Gary
Algorithm Updates | | gazza7770 -
I need help with drastic SERP difference between Bing and Google
One of our sites that has been around for a couple of years has about 60,000 pages showing on google, however, bing only shows 90 pages for the site. This same phenomenon has been happening across the board for our sites. Any ideas to improve our indexing results for bing?
Algorithm Updates | | atuomala0 -
When Google crawls and indexes a new page does it show up immediately in Google search - "site;"?
We made changes to a site, including the addition of a new page and corresponding link/text changes to existing pages. The changes are not yet showing up in the Google index (“site:”/cache), but, approximately 24 hours after making the changes, The SERP's for this site jumped up. We obtained a new back link about a couple of weeks ago, but it is not yet showing up in OSE, Webmaster Tools, or other tools. Just wondering if you think the Google SERP changes run ahead of what they actually show us in site: or cache updates. Has Google made a significant SERP “adjustment” recently? Thanks.
Algorithm Updates | | richpalpine0