Google Webmaster indicates robots.text access error
-
Seems that Google has not been crawling due to an access issue with our robots.txt
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent.A quick test shows I can access the file.
This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8%
Note the above message says 'over the last 24 hours', however the date is Jan-17
This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us.Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal.
Thanks!
-
Unfortunately, no. Free product = No Service
-
Cody, thank you.
And... there is no 'Google contact' for these kinds of issues, correct? (dumb question).
-
This happens from time to time on sites that I manage. I usually just check to make sure that it has been crawling since that date. If it has, then life is good.
If it hasn't, then I usually shout obscenities before trying to find the problem. The obscenities don't directly contribute to a solution, but it does seem to make me calm down a bit. Anyway, I digress.
Take a look at your webmaster tools, and see how well the site has been being crawled since that date. Is it all O.K.? If not, try to describe any changes you made in the weeks prior to that date.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I am tracking my domain e.g www.xyz.com to an existing google analytics account . I want to track www.xyz.com/blog separately in the google analytics account. How can I do that?
I am tracking my domain e.g www.xyz.com to an existing google analytics account . I want to track www.xyz.com/blog separately in the google analytics account. How can I do that?
Reporting & Analytics | | Windlass0 -
Is it possible to import data from an old Google Analytics profile to a new Google Analytics profile?
We have encountered a situation where a client's old SEO firm is refusing to grant us Admin access to our client's existing GA account. For security purposes (so the other SEO firm doesn't delete the existing GA profile) we have started a new Google Analytics profile. Again we do have access to the data in the old account. Is it possible to migrate this old data over (if we just have user access)? Thanks for the help
Reporting & Analytics | | RosemaryB0 -
Does Google Analytics track conversions in real time?
Looking at the section: Conversions > Goals > Overview - does Google Analytics provide real time / same day metrics for this conversion data or does it take 24 hrs?
Reporting & Analytics | | Rich_Coffman0 -
E-commerce data import Google Analytics
Hi there, Since a few weeks we have started a cooperation with a big online wholesale company which now sells our products. I want to import simple E-commerce data: amount of transactions and revenue. More detailed data such as amount of products, name of products, etc. is not necessary in the first place. Now I discovered the 'data import' functionality in Google Analytics but I can't find any suitable option for E-commerce data import. The data from the wholesale company is gathered in a nice export in a separate system, so there should be a way ti import this data into Analytics. Could anybody help me with this? Any advice is welcome! Thanks in advance.
Reporting & Analytics | | MarcelMoz
Marcel0 -
Webmaster Tools V. OSE Backlinks to Disavow
Hey Everybody, So I am getting to the point where I need to disavow some backlinks. I am noticing a discrepancy between the OSE backlinks and the GWMT backlinks. My question is do I put both sets of links in the Disavow tool? Just the links from GWMT? Just the links from OSE? Etc... Etc... Thanks Everybody!
Reporting & Analytics | | HashtagHustler0 -
Google analytics vs Webmaster tools data
Hi Which is more accurate WT or GA data ? Since GA reporting a KW (thats very recently fallen from page 1 to 3 hence looking into data to find the cause) in the Organic part of Search tab as having generated just 1 visitor over a month (hence presuming fall could be due to low visits from a page 1 result) whilst under Search Engine Optimisation tab (data sourced from WT i think) its reporting 5 click thrus from 150 impressions over same period resulting in a quite good 3.33% CTR (hence wouldn't expect to be the cause of a fall) and what i would have thought GA would report as 5 visits instead of the 1 they do report !? The reason im looking for answer in the data is because no on-page has changed and still scoring a grade A and off page metrics have all improved across the board (apart from small drop in majseo's Trust Flow) such as increased links, RD, Citation Flow, Ref Subnets etc etc etc Cheers Dan
Reporting & Analytics | | Dan-Lawrence0 -
If I install google Analytics, that can affect my SEO.?
I purchased a site that has a lot of Backlinks and when I started to make changes like installing google analytics! i suspect the performance in search engine was getting worst because of that, so I removed.
Reporting & Analytics | | pparma
Now I'm trying MOZ and needs GA, should I install it again??
Maybe it analyces the backlinks and tells google, I don't know, is that possible?0 -
Weird info from google analytics?
Hi Could anyone explain what these visits are in Google Analytics? Under traffic sources and organic I am seeing lots of entries with data like below. Any ideas what kind of traffic this is? Is it a bot and if so what is their purpose of it and is it recommended that you block it? Pages/Visit 1.00 Avg. Time on Site 00:00:00 % New Visits : 100% Bounce Rate: 100.00% Many Thanks
Reporting & Analytics | | ocelot0