Google Webmaster indicates robots.text access error
-
Seems that Google has not been crawling due to an access issue with our robots.txt
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent.A quick test shows I can access the file.
This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8%
Note the above message says 'over the last 24 hours', however the date is Jan-17
This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us.Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal.
Thanks!
-
Unfortunately, no. Free product = No Service
-
Cody, thank you.
And... there is no 'Google contact' for these kinds of issues, correct? (dumb question).
-
This happens from time to time on sites that I manage. I usually just check to make sure that it has been crawling since that date. If it has, then life is good.
If it hasn't, then I usually shout obscenities before trying to find the problem. The obscenities don't directly contribute to a solution, but it does seem to make me calm down a bit. Anyway, I digress.
Take a look at your webmaster tools, and see how well the site has been being crawled since that date. Is it all O.K.? If not, try to describe any changes you made in the weeks prior to that date.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Analytics Numbers Are Weird
Hi, I'm having a strange problem from past one month. My website gets about 10k pageviews a month with an average bounce rate of 50%. Lately, I observed a strange thing. The avg time on page of visitors from selected countries is less than 1 second and the bounce rate is zero!! How is this possible? This is happening from past one month and it would be really helpful if you guys could tell me what exactly is happening?? Ive attached a screenshot for better understanding. website url : https://www.specbee.com/ fIAXXgF 3zQuBAN
Reporting & Analytics | | ganesh10 -
Google Analytics & Google Sheets
Hi guys, I'm looking to use google analytics google sheets extension to pull data into a google sheet with metrics from other tools e.g. semrush, moz, etc. From my understanding with google analytics google sheet extension its designed to run reports and doesn't really allow you to add other metrics. Basically what i'm trying to do is this: https://www.useloom.com/share/c1e42bfa60bd46fca2b1120018969ce8 Any suggestions/advice on how to do this would be great. Cheers.
Reporting & Analytics | | jayoliverwright0 -
Google Analytics SEO Queries Not Showing
Hi All, This might be a silly question, but for all the properties I monitor in Google Analytics, I'm now showing no data for SEO Queries under Acquisition for the past 6 days. Normally I would expect a few day delay in queries, but nothing for 6 days is somewhat peculiar especially as it was functioning fine prior to November 12th. Does anyone have insight into what might be going on? Thanks! URaNMa3
Reporting & Analytics | | amichaels0 -
Bing Webmaster Tools data discrepency on traffic
Hi, For a given time period in Bing Webmaster Tools it says that combined clicks from Bing and Yahoo organic search are about 4,700 total, for the same time period in Google Analytics, combining sessions for Bing and Yahoo Organic comes out to about 8,900 total. Has anyone else experienced this discrepancy? Is this common? How can I get 8900 visits from 4700 clicks?
Reporting & Analytics | | IrvCo_Interactive0 -
Roger says I have 378 404 errors
Hi The report shows that I have 404 errors in the about of almost 300 but they appear to me to be active and part of the site. ??? Examples: NOTE: warning - these pages discuss BDSM and have some nude images here are some on the list http://bestslavetraining.com/groups/discipline-and-punishment/?acpage=6 http://bestslavetraining.com/groups/general-b-d-s-m/?acpage=13 also here is one that does not work http://bestslavetraining.com/activity/activity/ WHY does it show 404 errors for urls that work??? what am I missing and what in the world am I to correct if they appear on the SEOmoz error log.???
Reporting & Analytics | | Cuffsmaster0 -
Google Analytics - New account creation broken
Is it just me or is the new Google Account Creation button not working. After i set it up, it just simply doesn't save.
Reporting & Analytics | | TheGrid0 -
Omniture vs Google Analytics
What's your opinion on analytics tools? Specifically Omniture vs Google Analytics. Is Omniture really that much more powerful than GA? Have you used GA Premium, the enterprise package? My main question - Can GA (free or premium) do everything that Omniture can? If GA Premium and Omniture were the same price, which would you choose? Is one harder to implement, pull reports, tweak, ect? Thanks in advance.
Reporting & Analytics | | akim260 -
Google Analytice for Shopping Results
We are trying to optimise for Google Shopping and therefore would like to be able to monitor traffic to site from this source? How can this be done in Google Analytics? Thanks
Reporting & Analytics | | seanmccauley0