Google Webmaster indicates robots.text access error
-
Seems that Google has not been crawling due to an access issue with our robots.txt
Late 2013 we migrated to a new host, WPEngine, so things might have changed, however this issue appears to be recent.A quick test shows I can access the file.
This is the Google Webmaster Tool message: http://www.growth trac dot com/: Googlebot can't access your site January 17, 2014 Over the last 24 hours, Googlebot encountered 62 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 8.8%
Note the above message says 'over the last 24 hours', however the date is Jan-17
This is the response from our host:
Thanks for contacting WP Engine support! I looked into the suggestions listed below and it doesn't appear that these scenarios are the cause of the errors. I looked into the server logs and I was only able to find 200 server responses on the /robots.txt. Secondly I made sure that the server wasn't over loaded. The last suggestion doesn't apply to your setup on WP Engine. We do not have any leads as to why the errors occurred. If you have any other questions or concerns, please feel free to reach out to us.Google is crawling the site-- should I be concerned? If so, is there a way to remedy this? By the way, our robots file is very lean, only a few lines, not a big deal.
Thanks!
-
Unfortunately, no. Free product = No Service
-
Cody, thank you.
And... there is no 'Google contact' for these kinds of issues, correct? (dumb question).
-
This happens from time to time on sites that I manage. I usually just check to make sure that it has been crawling since that date. If it has, then life is good.
If it hasn't, then I usually shout obscenities before trying to find the problem. The obscenities don't directly contribute to a solution, but it does seem to make me calm down a bit. Anyway, I digress.
Take a look at your webmaster tools, and see how well the site has been being crawled since that date. Is it all O.K.? If not, try to describe any changes you made in the weeks prior to that date.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Webmaster Tools Average keyword position
Hello! For the last month i have notices a drastic change in Avg.position for a keyword on my clients site http://livoart.lv keyword: "kapu pieminekļi". According to webmaster tools it went from 13 to 4-2. The problem is that in orgranic search i keep seeing the keyword on the second page and the CTR of the keyword did not change at all. Why would webmaster tools show false data?
Reporting & Analytics | | AlexElks0 -
Webmaster Tools vs. Google Trends data doesn't add up
I am investigating a two-month 25% drop in organic traffic from Google to a client's site. When I turned to the Webmaster Tools data for the site, there is a clear, gradual drop over the course of a couple months both in impressions and clicks. In general, the drop occurred across many pages and for a large number of queries; there wasn't a core group of keywords or pages that saw the drop...it was more sitewide. Yet, the average rankings reported by WMT were, for the top 100 or so landing pages, not significantly different. The site hosts information about medical conditions, and I wouldn't expect any time-related variations in search volume, and this was confirmed by looking at Google Trends data for a number of the top keywords. I started to look at the data by query for all the top keywords (all ranked in the top 10), and saw the following general trend: impressions were down, rankings stayed in the top 10, and Google Trends showed either flat or rising volumes. So I am trying to make sense of that. If the search volume trend did not decline and rankings held inside the top 10, then how could the number of impressions drop significantly? Am I trusting the WMT data too much? But the reality is that the volume of traffic measured by Google Analytics from Google organic did indeed drop the way Webmaster Tools show it.
Reporting & Analytics | | WillW0 -
Google analytics - landing pages
Hello, I have a website and it contains landing pages. after tracking and analyzing the data in Google Analytics , I found out that the traffic for the whole site is affected by the visits to the landing page. I mean, the bounce rate is less than one minute. this is because users fill their details in the landing pages and leave my website. to improve my website, I need clear data about the traffic, bounce rate etc. in my website. I also need can you help me solve this problem I thought about create another account in Google Analytics and set it to my websites pages (not including the landing pages). and keep the current code for the landing pages. my question is regarding this: Is it legal to use 2 different GA code (each one of them belong to different account) in the same domain ? can you provide me with more information about Multi Account in Google Analytics, and how can I use it to divide the traffic in my website between the traffic and the data for the landing pages and for the website itself ?
Reporting & Analytics | | JonsonSwartz0 -
Google Images referral visits fell off a cliff
As of 28-Jan, referral traffic from Google images (/imgres) to my domain has pretty much vaporized. Visits are down 85%. It's actually not a disaster because most of those visits were from poorly optimized alt tags that were resulting in low quality visits. The interesting thing is that visit duration is up 80% during the same period. So I'm asking this question out of curiosity more than anything. Is this likely an algorithm tweak? I can't think of any major changes on my end. There's only one other data point to mention but I don't see how they'd be connected. I did two PRWEB releases on 29-Jan and 30-Jan that resulted in a few hundred new no-follow links back to my site. 22b7k2.png
Reporting & Analytics | | JonDiPietro0 -
How Google handles keywords and punctuation?
How does Google handle punctuation of keywords within its search? If I have three keywords together, but on the site there is a question mark between them, does Google ignore the question mark? Or should I think of a new way of writing the content so Google likes it more? And would this apply with articles (a, the, etc) and prepositions such as (in, of, etc.)? Thanks a lot.
Reporting & Analytics | | Rocket.Fuel1 -
Tracking PDF Downloads in Google Analytics
Hi, I work on a site that allows users to download whitepapers after filling out a form. Once they do this they are redirected to a URL which is the PDF. We use Wordpress and these documents were uploaded to the media center. I've tried researching how to track these downloads in GA, since the code is not present on these pages, but have read a few different answers. Anyone have firsthand experience? Thanks!
Reporting & Analytics | | tinarose0 -
How to measure number of visits from Google News coming from Google Universal Search (NOT referral coming directly coming from news.google.com) with google analyitcs
I'm running a news site, and I have a problem of accuratly measuring which traffic is REALLY coming from google news. I analyzed a lot of individual articles and I come to the conclusion, that the visits, that come from the google news section in the universal search results are counted as "normal" search engine traffic in google analytics. So if you do a Google search for a topic that includes links from Google news, you don't get an accurate referral count. As an example, if you do a search for "eBay", incorporated into the page 1 search results you may also see Google news results as well.
Reporting & Analytics | | Mulle
If someone clicks on that Google news link that appears in Google search, it shows up in Google analytics as a referral from Google search, when it was actually from a Google news referral. I was already checking google analytics and google news help forums and searched SEO blogs for this. But I wasn't able to find a working solution. Can anybody help me out with this problem? Thanks so much, Matthias0