Google Webmaster indicates robots.text access error
-
Seems that Google has not been crawling due to an access issue with our robots.txt
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent.A quick test shows I can access the file.
This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8%
Note the above message says 'over the last 24 hours', however the date is Jan-17
This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us.Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal.
Thanks!
-
Unfortunately, no. Free product = No Service
-
Cody, thank you.
And... there is no 'Google contact' for these kinds of issues, correct? (dumb question).
-
This happens from time to time on sites that I manage. I usually just check to make sure that it has been crawling since that date. If it has, then life is good.
If it hasn't, then I usually shout obscenities before trying to find the problem. The obscenities don't directly contribute to a solution, but it does seem to make me calm down a bit. Anyway, I digress.
Take a look at your webmaster tools, and see how well the site has been being crawled since that date. Is it all O.K.? If not, try to describe any changes you made in the weeks prior to that date.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What about this (google crawl)?
Recently we did a serious effort on SEO with SEO Yoast (Wordpress). And after a few months of tweaking old articles we get this impact on crawl search.. Is this graph normal? s1TQgv9.png
Reporting & Analytics | | noodweerbenelux0 -
Google Analtyics during site redesign
Hi, We will be launching a new redesign for our website. There will be new URLs and navigation and almost everything (except for static pages like about and contact) will be different. The overwhelming opinion seems to say that it's important to keep the same Google Analytics profile. How can we compare the past URLs to the new ones if they are completely different. Does anyone have any experience in this? Did you create any segmentation? Thanks 🙂
Reporting & Analytics | | WSteven0 -
Google Analytics Not Tracking 100% of Visits?
Hi all, We're having an issue with Analytics where we are getting different figures from what Silver Pop are saying. For example email campaign A sent via Silver Pop, with Google Analytics tracking code show's 50 unique clicks in Silver Pop. Looking at Google Analytics there are only 10 visits from that campaign. So I thought it could be something with the tracking, but there wasn't a significant rise in web visits = either Google Analytics is not recording visits properly or Silver Pop figures are wrong. I'm more inclined to think that it's something to do with Google Analytics. Has anyone come across something similar? Where one system is showing you X amount of visits but the figures on Google Analytics don't add up? A few quick things already covered: Double checked the links have been tracked properly, but this doesn't explain the low increase in web visits generally We've double checked that Google Analytics tracking code is properly installed (and it is / was at the time of send). Any help would be much appreciated! Thanks guys.
Reporting & Analytics | | RKHStaff1 -
Google Analytics Not Working
I added the code before tag but still google not showing it is installed. Status: Tracking Not Installed Last checked: Mar 15, 2013 10:38:10 PM PDT Can someone check my domain - www.plugnbuy.com
Reporting & Analytics | | chandubaba0 -
Roger says I have 378 404 errors
Hi The report shows that I have 404 errors in the about of almost 300 but they appear to me to be active and part of the site. ??? Examples: NOTE: warning - these pages discuss BDSM and have some nude images here are some on the list http://bestslavetraining.com/groups/discipline-and-punishment/?acpage=6 http://bestslavetraining.com/groups/general-b-d-s-m/?acpage=13 also here is one that does not work http://bestslavetraining.com/activity/activity/ WHY does it show 404 errors for urls that work??? what am I missing and what in the world am I to correct if they appear on the SEOmoz error log.???
Reporting & Analytics | | Cuffsmaster0 -
Multiple Domains in Webmaster Tools
Our client has four domains all redirecting to one domain. aquatron.info, www.aquatron.info, and www.aquatronhardwatersolution.com are all set up to redirect to aquatronhardwatersolution.com. I would like to make sure that we are getting the most link juice and that they are set up correctly in Google Webmaster Tools. Any help would be great. Thanks!
Reporting & Analytics | | PolarisWebSEO0 -
Google Analytics - Referral Traffic Question
How Google Analytics determine that some particular web site referred traffic if there is no back link on that site?
Reporting & Analytics | | DiamondJewelryEmpire1 -
Google Analytics - Intelligence Custom Alerts
When you add an intelligence custom alert does it look at the history and backdate issues? I've added a few custom alerts, even basic ones like my homepage gets 5 views, create an alert. I either need to wait a day or something is going wrong. Cheers.
Reporting & Analytics | | Seaward-Group0