Google Webmaster indicates robots.text access error
-
Seems that Google has not been crawling due to an access issue with our robots.txt
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent.A quick test shows I can access the file.
This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8%
Note the above message says 'over the last 24 hours', however the date is Jan-17
This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us.Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal.
Thanks!
-
Unfortunately, no. Free product = No Service
-
Cody, thank you.
And... there is no 'Google contact' for these kinds of issues, correct? (dumb question).
-
This happens from time to time on sites that I manage. I usually just check to make sure that it has been crawling since that date. If it has, then life is good.
If it hasn't, then I usually shout obscenities before trying to find the problem. The obscenities don't directly contribute to a solution, but it does seem to make me calm down a bit. Anyway, I digress.
Take a look at your webmaster tools, and see how well the site has been being crawled since that date. Is it all O.K.? If not, try to describe any changes you made in the weeks prior to that date.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Analytics Not Properly Attributing Goals
My GA has been working fine for years and is now suddenly attributing 100% of my goals/conversions to a "referral" source. In this case "crm.zoho.com". (see attached url) Our contact form is a zoho CRM form. When prospect fills it out online, it dumps them to a thank-you page and the conversion is counted. That all works just fine, but it is not attributing the conversion to Organic or PPC or Direct as it was a month or so ago. I'm not sure if this may be the cause, but I was cleaning up GA about a month ago, deleting some filters I didn't think I needed any more. Thank you for your help. 9CRUx
Reporting & Analytics | | sanctuary2420 -
How To Stop Google's "Fetch & Render" From Showing Up In Google Analytics
Hi all, Within Google's "Fetch & Render" (found in Google Search Console) is the ability to index certain pages from my website on-demand. Unfortunately, every time I ask Google to index a page, it registers as a bounce in Google Analytics. Also, if it means anything, my website (www.knowtro.com) is a single-page application, functioning similarly to Google. If you guys know of any solution to this problem, please help! I originally thought that Google would know to block its own Fetch & Render crawler from Google Analytics but that doesn't seem to be the case. Thanks, Austin
Reporting & Analytics | | A_Krauss0 -
Google Analytics unexplainable traffic spike
Viewing this report **Behaviour > Site content > All pages **with primary dimensions of **Page ** and **Page title **are showing different results for one specific page on our site. We noticed a huge spike in pageviews and entrances to that specific page on our site. The user flow report shows traffic going through and from the page (in high volume) from other pages on the site which suggests that it's not the usual case of spam referrals. When I view the report above by page, I get unrealistic data. Over 600% increase in pageviews and over 900% increase in entrances. When I view by page title however, I get realistic results. Can anyone help shed some light on why these two reports will be different? Anyone else seeing similar issues?
Reporting & Analytics | | OptiBacUK0 -
I want to take down some pages, how do I inform google?
Hey Guys, I'm hoping someone can help - I'm in the midst of a site re-design - whilst one of our biggest reasons for the redesign was to create more space to write valuable content and unique content I have been reading other posts on moz about content auditing. I have come across a few articles in my own blog that are from 250 - 300 words to which the articles seem similar and the traffic is low. I'm wondering while I'm left to consolidate these articles and create a fresh article for each entry that is more in-depth, when I consolidate or delete these pages - do I need to inform google these pages have now been deleted? If so using Wordpress what is the best way to do this? Cheers and would appreciate some advise Thanks
Reporting & Analytics | | edward-may0 -
Moz is showing different "errors" than Webmaster tools
I have set up my Moz campaign and the crawl errors are showing multiple duplicate content and page titles however when I check my webmaster tools data, these errors are not showing up. Is this normal and who should I listen to?
Reporting & Analytics | | LabelMedia0 -
Use Google Spreadsheet for Top Keywords Report
I've been using the super-awesome Google doc described here http://www.seomoz.org/blog/custom-reporting-using-google-analytics-and-google-docs for all kinds of reporting and I've customized it quite a bit. But I can't figure out how to feed text-based fields from GA into this doc. For example, can I make it so this doc spits out top 10 keywords? Thanks!
Reporting & Analytics | | Oxfam_America0 -
301 Tracking with Google Analytics hurts SEO?
Hey guys, I have an old-domain.com and a new-domain.com. old-domain.com has reputation and through a 301 redirect it's handling it over to new-domain.com. I want to know how many people still visit old-domain.com and track this with Google Analyitcs. I read the best solution for this would be a Google Campaign URL. My question: Does a URL like "http://www.new-domain.com /?utm_source=&utm_medium=redirect&utm_campaign=" hurt the effectiveness of the 301 redirect?
Reporting & Analytics | | optimiert-es0 -
Google Analytics - Intelligence Custom Alerts
When you add an intelligence custom alert does it look at the history and backdate issues? I've added a few custom alerts, even basic ones like my homepage gets 5 views, create an alert. I either need to wait a day or something is going wrong. Cheers.
Reporting & Analytics | | Seaward-Group0