Google Webmaster indicates robots.text access error
-
Seems that Google has not been crawling due to an access issue with our robots.txt
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent.A quick test shows I can access the file.
This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8%
Note the above message says 'over the last 24 hours', however the date is Jan-17
This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us.Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal.
Thanks!
-
Unfortunately, no. Free product = No Service
-
Cody, thank you.
And... there is no 'Google contact' for these kinds of issues, correct? (dumb question).
-
This happens from time to time on sites that I manage. I usually just check to make sure that it has been crawling since that date. If it has, then life is good.
If it hasn't, then I usually shout obscenities before trying to find the problem. The obscenities don't directly contribute to a solution, but it does seem to make me calm down a bit. Anyway, I digress.
Take a look at your webmaster tools, and see how well the site has been being crawled since that date. Is it all O.K.? If not, try to describe any changes you made in the weeks prior to that date.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Analytics, Wordpress, Sumo - who to trust
Something is wrong. I have a website that according to Google Analytics gets zero relevant clicks, with most of the clicks it does get coming from Russia. Wordpress says it gets a hundred or so visitors a month, with most traffic going to the pages that I know rank. Sumo says the heat map got 76 clicks. How would you figure out what's really going on?
Reporting & Analytics | | julie-getonthemap0 -
How to Configure Google Analytic API?
Hello All, I want to implement google analytic api can any one show me the whole process? Regards, Mit
Reporting & Analytics | | mit0 -
How do you adjust the Google Analytics date range?
Hello! Every time I log into Google Analytics (via desktop), the date range is set to October 1, 2013 to October 28, 2013. I've looked throughout the GA settings to find a place where I change this default setting to anything else ("Last week?") but have not be able to find where that setting is. Can you help? Losing my mind,
Reporting & Analytics | | SmileMoreSEO
Erik0 -
Webmaster Tools vs. Google Trends data doesn't add up
I am investigating a two-month 25% drop in organic traffic from Google to a client's site. When I turned to the Webmaster Tools data for the site, there is a clear, gradual drop over the course of a couple months both in impressions and clicks. In general, the drop occurred across many pages and for a large number of queries; there wasn't a core group of keywords or pages that saw the drop...it was more sitewide. Yet, the average rankings reported by WMT were, for the top 100 or so landing pages, not significantly different. The site hosts information about medical conditions, and I wouldn't expect any time-related variations in search volume, and this was confirmed by looking at Google Trends data for a number of the top keywords. I started to look at the data by query for all the top keywords (all ranked in the top 10), and saw the following general trend: impressions were down, rankings stayed in the top 10, and Google Trends showed either flat or rising volumes. So I am trying to make sense of that. If the search volume trend did not decline and rankings held inside the top 10, then how could the number of impressions drop significantly? Am I trusting the WMT data too much? But the reality is that the volume of traffic measured by Google Analytics from Google organic did indeed drop the way Webmaster Tools show it.
Reporting & Analytics | | WillW0 -
Regular Expressions in Google Analytics
Hi All I've been struggling to create a regular expression for a Google Analytics goal step that would match the following: ^/specifictextstring/anytextstring/anytextstring/
Reporting & Analytics | | Cabbagefeet
^/specifictextstring/anytextstring/ However I don't want it to match any URLs that end with: /anytextstring**_**phonecall or /phonecall, for example: /specifictextstring/anytextstring/anytextstring/anytextstring**phonecall
/specifictextstring/anytextstring/anytextstring**phonecall
/specifictextstring/anytextstring/anytextstring/phonecall
/specifictextstring/anytextstring/phonecall Does anyone have any ideas? Thanks in advance for all contributions.0 -
Pinterest links have been growing in Webmaster Tools
I've been using Pinterest for a couple of years now. It has been great in promoting our brand and generating referral traffic. My question is this. I've noticed in Webmaster Tools, in the "Links to your site" section, that my links from Pinterest has really shot up.It's gone from 3,500 links, 6 months ago, to over 23,000 recently. Those of you using Pinterest, have you seen a similar increase? Just wondering if Google is recognizing more of the links, even though they are "no-follow" links.
Reporting & Analytics | | tdawson090 -
301 redirect visible on Google.
Hello, This is a strange one for me. We have 301 redirected our .com domain to the .co.uk domain. The strange thing is that if you gogole the .com domain (theloanengine.com) there is a result with the website description. If you click on the result you are redirected tot he website homepage. One other thing: I've discovered this 301 issue because Google Analytics started to show me a few days ago referrals from the .com domain. I don't know if these things are connected. Cornel
Reporting & Analytics | | Cornel_Ilea0 -
Google Analytics should track users from iPhone App
Hi Mozzers, you say there are no dumb questions, here is one. We've build an iPhone App for our Website working-dog.eu. So users can read there new messages with ease when they are out of the office, read about breeds, ... do all the nice stuff they want. I don't think so, but is it possible to track these users with Google Analytics? When the come via App, the Analytics code will not run for them. But maybe it could start an AJAX request or something like this? But how should the code run, without a browser which can interprete JavaScript? So, in short: Is it possible to track users from iPhone App with Google Analytics? Kind regards Patrick
Reporting & Analytics | | mdoegel0