Google Webmaster indicates robots.text access error
-
Seems that Google has not been crawling due to an access issue with our robots.txt
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent.A quick test shows I can access the file.
This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8%
Note the above message says 'over the last 24 hours', however the date is Jan-17
This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us.Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal.
Thanks!
-
Unfortunately, no. Free product = No Service
-
Cody, thank you.
And... there is no 'Google contact' for these kinds of issues, correct? (dumb question).
-
This happens from time to time on sites that I manage. I usually just check to make sure that it has been crawling since that date. If it has, then life is good.
If it hasn't, then I usually shout obscenities before trying to find the problem. The obscenities don't directly contribute to a solution, but it does seem to make me calm down a bit. Anyway, I digress.
Take a look at your webmaster tools, and see how well the site has been being crawled since that date. Is it all O.K.? If not, try to describe any changes you made in the weeks prior to that date.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I get a - 'Temporarily unreachable' error message when I 'Fetch as Google' Any ideas please??
I wanted to Fetch this page and got this error from Google - Temporarily unreachable. I've never had this issue before?? I checked another page and it came back as 'Complete', so no problems there? Any ideas? Thank you in advance.
Reporting & Analytics | | MissThumann0 -
Tasks for Google Analytics training
Hi Mozzers, I'm delivering some Google Analytics (Fundamentals level) training, and trying to make it was fun and as interesting as possible... which is quite a challenge when it comes to GA. I was just wondering if you're aware of training tasks, or interactions, I could bring into this kind of training session? The group are particularly interested in user journeys and the effectiveness of content. Thanks!
Reporting & Analytics | | A_Q0 -
Google News Bug
During today and yesterday we are getting a lot of visits (Over 400) from http://news.google.com the visits are all at the same time and to different pages, the visits are very brief so much so that it pushes our bounce rate up to 69% in GA. Is there anything we can do to stop this? Thank you in advance?
Reporting & Analytics | | Palmbourne0 -
Setting up goals - google analytics
Hi Im new user google analytics I would like to set up goal for website. I asked to basic questions: What are my bussiness objectives? Answer:make the visitor to click on a advertisment (affiliate links and affiliate banners) to be redirected to company website to finish purchase. How to set up goals? How can I track ads (affiliate banners and affiliate links) on my website?
Reporting & Analytics | | info_tipovanie-stavkovanie.com0 -
2 questions on avoiding issues with Google and while being right in it.
Hi SEOmoz community In fact I have two questions I would like to ask (with future SEO in mind). Do you consider a WordPress Multisite or various Single installs 'safer' for SEO? Theoretically, having various sites packed into one Multisite network seems like an ideal solution. However, is there a chance that once a site in the network encounters a little 'negative turbulence', that your other sites in the network might get impacted too due to the cross-referencing, linked account i.e. Webmaster Tools etc.? It would seem outrageous, but then again I wouldn't rule it out. Do I even have to go as far as setting up new Gmail, Google Analytics and Webmaster Tools accounts, so they (the sites) are technically not linked? You can see, I don't trust search engines one bit... Is there still a point posting articles once Google is having a hissy fit with your site? Basically I am currently going through a 'rankings and traffic drops storm'. It's not as bad as being de-indexed, but it's still having enough of an impact. In addition, Google does not seem to treat my new articles (unique content) with the same attention anymore i.e. does not seem to index them 'fully' or not at all (i.e. posting the headline in Google should return the article, but it doesn't). Is there even a point spending time now and posting new material or may it pick it up again once I am through this low phase? Does Google still index what it considers worth or is it a waste of time right now to keep posting, posting and posting more? Thanks for your help. I really appreciate it.
Reporting & Analytics | | Hermski0 -
Is Google Webmaster Tools Index Status Completely Correct?
So I was thrilled when Webmaster tools released the little graph telling you how many pages are indexed. However, I noticed with one of my sites, that it was actually going down over the past year. I've only been doing the SEO on the site for a few months, so it wasn't anything I did. The chart is attached to this post. Now here's the funny part. I haven't really noticed anything out of the ordinary for keyword ranking dropping off. I also tested my most recent page that I put up. 3 days after I posted it, I could find it in a Google search. Shouldn't this mean that's it's indexed? I can also find any other page I've posted in the last few months. Another oddity is that I submitted a sitemap a while ago when the site was only 22 pages. The sitemap index count says 20 of those pages are indexed. The chart only says that there are 3 indexed pages right now. However, I can clearly find dozens of pages in Google searches. Is there something I'm missing? Is my chart for this website broken? Should I report this to Google? Has anyone had a similar issue? decreaseIndex.png
Reporting & Analytics | | mytouchoftech0 -
Backlinks on Google Webmaster Tools
As I was reviewing my Google Webmaster tools, I notices a major drop in the number of back links. It used to show over 4,500 links to my site yet the other day it dropped down to 41. None of the directories I've submitted to are showing, nor any blog comments I've posted. Since then, my SEO traffic has started to drop for some keywords. Anyone know why?
Reporting & Analytics | | MikeAndres0 -
Mobile and Google Analytics
As my mobile numbers are going up in Google Analytics I'm trying to figure out what phone runs what browser. I see that 93% of mobile users have Safari which I believe is iPhone, iPad and iPods. There is a 6% usage of "Mozilla Compatible Agent". What cell phone would this be?
Reporting & Analytics | | LabadieAuto0