Google Unable to Access Robots.txt
-
We haven't made any changes to the robots.txt file and suddenly Google claims they can no longer access the file.
The site has been up and active for well over a year now. What are my next steps?
I have included a screenshot of the top half of the file. See anything wrong?
-
Would you mind sharing the solution - if not related to domain intricacies
Curious to know - only for knowledge purpose
Many thanks
-
Thanks Phillip. I think we're on the way to discovering the issue.
I certainly appreciate the feedback!
-
I think you should allow general access before beginning to disallow various sections below:
Allow: /
Though semantically, I think your robots.txt is in order. But if Google says they cannot acces robots.txt it seems like they cannot physically access the file itself on yourdomain.com/robots.txt, not that they cannot crawl the page due to your robots.txt.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google's stand on LSI keywords?
Hi all, So the keywords which appear while typing some keywords and suggested keywords at the bottom of the search results page are refereed as LSI keywords. I been noticing some of the LSI keywords for years related to our industry and Google now suddenly changed them. I wonder why it would be. I can see competitors are started using those LSI keywords widely, is that the reason Google changed them? Thanks
Algorithm Updates | | vtmoz0 -
Google ranking impact: Returning visitor vs New visitor
Hi all, If a website's traffic increase in "New visitors"; will this impact rankings? Do the website overall traffic affect rankings? How much this is related with ranking improvement for main keywords? Just because thousands of visits increased for website, will it count as a strong ranking improvement signal? Thanks
Algorithm Updates | | vtmoz0 -
Does Google like pricing information?
Over the last year I have noticed a trend in a couple of industries. Google seems to prioritise landing pages with pricing information in the content. This seems more important than it used to. One industry is high end industrial machines. Traditionally there isn't a price list as everything is bespoke for the customer. Low end machines that display an off the shelf price are now ranking higher than they used to. This is frustrating because the different machines meet different customer requirements. However, both sorts of customers are likely to use the same search terms. Has anyone else noticed this trend?
Algorithm Updates | | Brighton-Soundsystem0 -
Drop in Traffic from Google, However no change in the rankings
I have seen a 20% drop in traffic from google last week (After April 29th). However when I try to analyze the rank of the keywords in the google results that send me traffic they seem to be the same. Today (6th March) Traffic has fallen further again with not much/any visible change in the rankings. Any ideas on what the reason for this could be? I have not made any changes to the website recently.
Algorithm Updates | | raghavkapur0 -
Profile Picture Display Next to Map on Google Search
On a recent Google search, I noticed that the top result also displayed the Google map of the location next to it but aside from that, it also displayed that business' Google+ Profile picture. I'm wondering how that is done. We have a Google+ account as well with an image associated with our business. At this point, the map doesn't even display for our business, but how does one get the map and the profile picture to appear on search results? Here's a link to the search result in question:
Algorithm Updates | | atuomala
http://www.google.com/search?q=tech+squad&rlz=1C1CHFX_enUS511US511&aq=f&oq=tech+squad&sourceid=chrome&ie=UTF-8#hl=en&rlz=1C1CHFX_enUS511US511&sclient=psy-ab&q=tech+squad+wi&oq=tech+squad+wi&gs_l=serp.3..0l2j0i30l2.6041.6590.0.6592.3.3.0.0.0.0.79.158.2.2.0.les%3B..0.0...1c.1.4.psy-ab.DoA-PSlyt-A&pbx=1&bav=on.2,or.r_gc.r_pw.r_qf.&bvm=bv.42768644,d.cGE&fp=b5d841bb1ab219dc&biw=1440&bih=785 tech_squad_serp.jpg0 -
Google Algo Update In Que. What consititues over optimization?
http://www.pcmag.com/article2/0,2817,2401732,00.asp According to this, Google is bringing the hammer down soon on another 10-20% of the search results. While we don't advocate keyword stuffing, exchanging links, or anything too risky I am still concerned. Do we know if the example "perfectly optimized page"; http://www.seomoz.org/blog/perfecting-keyword-targeting-on-page-optimization is now going to be penalty bait? Is this over stuffing? Also, how might this effect ecommerce sites in particular?
Algorithm Updates | | iAnalyst.com2 -
Google +1 link on Domain or Page?
Since its release, I've seen Google +1 being used across an entire domain but only reference the root href in the code snippet. At the same time, you see other sites use +1 more naturally with the button being specific to the page you're on. What's your take on this? To clarfiy, do you add: or .. on each page.
Algorithm Updates | | noeltock0 -
CTR for Google Rankings
I run a local business, and I'm working on ranking for keyword + city. I currently rank on the first page for just about every keyword I'm working on, but only the top 3 for a little less than half. Because the search volume is so low for each keyword (for most cities Google doesn't have an estimated monthly search volume) the grand total of a few searches a month for each keyword + city combination is where I get my traffic. Although I seem to be getting consistently higher in the rankings, I am curious as to how much more traffic I can expect. I read somewhere that sites that are ranked number one are clicked 50% of the time, number two 20% of the time, number three 15% and from there on it goes down fast. Rank 7 and on is below 1%. Probably around 30% of my keywords are ranked between 7-10 and probably about 20% are ranked 4-6. Are the CTR numbers fairly accurate? I understand that there are a lot of influences on CTR, such as title/description, but generally is that somewhat accurate? If it is, I am missing out on A LOT of traffic. I am pulling about 800 unique visitors a month from Google. If I get in the top 3 for most of my keywords, can I expect significantly more traffic? I ask the question because there are many other things I could be doing with my time to help the business aside from SEO. I don't want to be working constantly on SEO if traffic is only going to increase very little.
Algorithm Updates | | bjenkins240