Google Webmaster indicates robots.text access error
-
Seems that Google has not been crawling due to an access issue with our robots.txt
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent.A quick test shows I can access the file.
This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8%
Note the above message says 'over the last 24 hours', however the date is Jan-17
This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us.Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal.
Thanks!
-
Unfortunately, no. Free product = No Service
-
Cody, thank you.
And... there is no 'Google contact' for these kinds of issues, correct? (dumb question).
-
This happens from time to time on sites that I manage. I usually just check to make sure that it has been crawling since that date. If it has, then life is good.
If it hasn't, then I usually shout obscenities before trying to find the problem. The obscenities don't directly contribute to a solution, but it does seem to make me calm down a bit. Anyway, I digress.
Take a look at your webmaster tools, and see how well the site has been being crawled since that date. Is it all O.K.? If not, try to describe any changes you made in the weeks prior to that date.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Strange 404 Pages Appearing in Google Analytics
Hi, My client has some strange urls appearing in GA which lead to error pages. Please see the following image: https://imgur.com/a/6TPO8yL e.g URLs like /h/6445738.html I've used screaming frog to see if these pages exist on the website and I can't find them, anywhere. Therefore how are they coming up in GA? If anyone could please help I'd really appreciate it.
Reporting & Analytics | | SolveWebMedia0 -
Alternative to Google Analytics
Hey Everyone, My company has just changed the order processing software we are using and it's causing some issues with Google Analytics conversion funnel tracking. Specifically, there is one point in the funnel where making certain selections (which about half the people do) causes the page to reload. Also, on the login/register page, if they miss a field, the software tells them missed the field, but loads a new page which has a different URL which is not a part of the funnel when a mistake like that isn't made. All of this is causing Google Analytics to report people as leaving the conversion funnel when they really haven't. About a third of the traffic is being shown as exiting the funnel with the exit URL being the exact same URL as the step they are supposedly exiting from (example: the visitor enters on page1, moves to page2, Google is showing that they exit on page2 and go to...page2. Does anyone have any suggestions of how to deal with this in Google Analytics? If not, do you have a recommendation of an alternative analytics program which can deal with the situations mentioned above? PS - Changing the way the checkout software works does not appear to be a viable option. Kurt Steinbrueck
Reporting & Analytics | | Kurt_Steinbrueck
OurChurch.Com1 -
Testing IP Exclusion Filters in Google Analytics
(I hope I haven't missed something simple 😉 I have add IP exclusion filters in Google Analytics. What I haven't been able to determine is how to test to see if it is working. I'd imagine there would be a dip in traffic but if a site is low volume it might not even register. Thanks!
Reporting & Analytics | | AUDigitalMarketing0 -
Google Places Account
Hi, I'm doing Google maps optimization for my local clients, so she uses same login for personal Gmail and Google places, is there a way to change a password for Google places, without giving away her password that she uses for personal Gmail? Thanks for help.
Reporting & Analytics | | tonyklu0 -
Enabling Webmaster Tools data within Analytics
Hello, Im having a hard time connecting webmaster tools within Google Analytics i want to be able to see search queries in GA That's what Google tells me to do : "You can visit the Property Settings page in Analytics account management to change which of your Webmaster Tools sites' data you wish to show, and control which profiles on your Web Property have access to view the data." I cant find "Property Settings Page" in google analytics, or anything that has to do with "Webmaster tools" I was wandering if you can help me on that 🙂 Thanks
Reporting & Analytics | | tonyklu0 -
500 errors and impact on google rankings
Since the launch of our newly designed website about 6 months ago, we are experiencing a high number of 500 server errors (>2000). Attempts to resolve these errors have been unsuccessful to date. We have just started to notice a consistent and sustained drop in rankings despite our hard sought efforts to correct. Two questions... can very high levels of 500 errors adversely effect our google rankings? And, if this is the case, what type of specialist (what are they called) has expertise to investigate and fix this issue. I should also mention that the sitemap also goes down on a regular basis, which some have stated is due to the size of the site (>500 pages). Don't know if they're part of the same problem? Thanks.
Reporting & Analytics | | ahw0 -
Omniture vs Google Analytics
What's your opinion on analytics tools? Specifically Omniture vs Google Analytics. Is Omniture really that much more powerful than GA? Have you used GA Premium, the enterprise package? My main question - Can GA (free or premium) do everything that Omniture can? If GA Premium and Omniture were the same price, which would you choose? Is one harder to implement, pull reports, tweak, ect? Thanks in advance.
Reporting & Analytics | | akim260 -
How to combine several subdomain in google Analytics
Good afternoon. I have a question.
Reporting & Analytics | | meteorr
I have a main site site.com + many subdomains example1.site.com, example2.site.com, example3.site.com, example4.site.com .... I need to combine google Analytics reports into one. Information from: example2.site.com, example3.site.com
showed in example1.site.com Yk64j0