Do I have a robots.txt problem?
-
I have the little yellow exclamation point under my robots.txt fetch as you can see here- http://imgur.com/wuWdtvO
This version shows no errors or warnings- http://imgur.com/uqbmbug
Under the tester I can currently see the latest version. This site hasn't changed URLs recently, and we haven't made any changes to the robots.txt file for two years. This problem just started in the last month. Should I worry?
-
Today it has a green check mark, and absolutely no changes were made to the website since I asked this question.
-
It could be that your server had a hard time when Google tried to view your robots.txt file that's why it wouldn't be able to fetch it. As long as this issue doesn't prevent Google anymore in the future it's not much to worry about.
-
That would make me feel more confident of a false error being reported. Time to closely monitor the crawl logs, look at server stats, and keep an eye on GWT for a change in the reporting/indexing. I would also go into the GWT forums and post, see if anyone is reporting a similar error these past couple days.
-
I can't post the domain but I know it is accessible.
When I go to the tester it shows the live robots.txt with no problems. I also can look at the server logs and see that it is being crawled, but being crawled less then Bing Crawls. Also the Bing Webmaster Tools is showing no problems.
-
Can you post your domain? Manually checking the robots.txt file would help.
I've checked many of my GWT accounts and I am not showing a sudden robots.txt error. It could be a false error, but I would take anything with the robots.txt file seriously. You'll want to make sure that it is in fact accessible to all the crawlers desired.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEO & IFrame problem
Hi All, I will try and keep this as simple as possible. My product page links to a separate page with an IFrame, giving my users the option to upload artwork for the product. The IFrame contains the external file upload site (mail big file). When finished, the user can use a button link to return to the product page to continue with their order. As soon as the page with the IFrame was crawled by Google, the IFrame page started to rank in place of where my product page used to rank, yet there is no content on the page relating to my product (just a file upload). So now users are visiting the IFrame via the same query which must be an absolute headache and not useful at all. I have tried the following: 1. Added a line in body text which contains an internal link pointing to the product page using exact match anchor text for the query. (This didn't work) 2. I applied a no index tag to the IFrame, and now my product page is no longer ranking at all. Can anyone help me solve this puzzle. I believe I might be missing something. Kind regards, Adam
Technical SEO | | SO_UK0 -
Google+ Authorship, Rich Snippits and Three Names - a Problem?
Hello All, I have a conundrum that I thought I'd resolved - but that's popped its gnarly old head over the parapet again. I have a number of websites that I'd like to have show my ugly Google+ mug as author in the Google SERPS. I jumped through all the authorship verification hoops that Google threw at me and I thought I'd won. The problem? I have three names: Nick Beresford-Davies. One example of a page that I'm trying to achieve authorship with is: http://www.graphic-design-employment.com/illustrator-how-to-make-a-pattern.html I have verified authorship of the above website on my Google Profile:
Technical SEO | | Tinstar
https://plus.google.com/u/0/107765436751760696335/about Originally I footed the page with Nick Beresford-Davies (hyphenated) and the Structured Data Testing Tool ignored the hyphen and just saw Nick Beresford. So I tweaked my online name (to please Google!) to Nick Beresford Davies (no hyphen). Initially this seemed to work - but I just checked again and now Google, for reasons only known to itself, sees "nick davies" as the author, completely ignoring the name in the footer of the page (by Nick Beresford Davies) and the fact that the site has been verified by Google+. This is also the case for all other websites that I contribute to - and not all the bylines are in the footer - some are by the headline. When I test pages on the structured testing tool and enter my Google+ profile, it replies: nick davies, we've found your name as one of the authors from the page. You can use "Authorship verification by email" method above to verify your authorship.Error: Author name found on the page and Google+ profile name do not match. Please consider adding markup to the site.Much as I would like to succeed on the Google SERPS, I draw the line at changing my name to keep this robot happy - so if anyone has any suggestions, or can see any obvious step that I've missed, I'd be very grateful. I find it hard to believe that no other double-barrelled website author exists - so I'm hoping I'm not the only one to have experienced this... Thanks!0 -
Robots.txt file
How do i get Google to stop indexing my old pages and start indexing my new pages even months down the line? Do i need to install a Robots.txt file on each page?
Technical SEO | | gimes0 -
Is there a reason to set a crawl-delay in the robots.txt?
I've recently encountered a site that has set a crawl-delay command set in their robots.txt file. I've never seen a need for this to be set since you can set that in Google Webmaster Tools for Googlebot. They have this command set for all crawlers, which seems odd to me. What are some reasons that someone would want to set it like that? I can't find any good information on it when researching.
Technical SEO | | MichaelWeisbaum0 -
Problem with 1 Domain but not 60 Others
Hi, I have about 60+ domains which are spread across a few different IP address's and a couple of Servers The domains use the same template, which is modified per the specific domain Out of the 60, a single domain attracts virtually no traffic. In google webmaster, there are no messages, nothing out of the order The supporting blogs etc... for that domain are all fine. Any idea why one domain does not perform, while the other 60 plug are successful?
Technical SEO | | Bucky0 -
Problem with indexed files before domain was purchased
Hello everybody, We bought this domain a few months back and we're trying to figure out how to get rid of indexed pages that (i assume) existed before we bought this domain - the domain was registered in 2001 and had a few owners. I attached 3 files from my webmasters tools, can anyone tell me how to get rid of those "pages" and more important: aren't this kind of "pages" result of some kind of "sabotage"? Looking forward to hearing your thoughts on this. Thank you, Alex Picture-5.png Picture-6.png Picture-7.png
Technical SEO | | pwpaneuro0 -
Mobile site - allow robot traffic
Hi, If a user comes to our site from a mobile device, we redirect to our mobile site. That is www.mysite/mypage redirects to m.mysite/mypage. Right now we are blocking robots from crawling our m. site. Previously there were concerns the m. site could rank for normal browser searches. To make sure this isn't a problem we are planning on rel canonical our m. site pages and reference the www pages (mobile is just a different version of our www site). From my understanding having a mobile version of a page is a ranking factor for mobile searches so allowing robots is a good thing. Before doing so, I wanted to see if anyone had any other suggestions/feedback (looking for potential pitfalls, issues etc)
Technical SEO | | NicB10 -
Subdomain Robots.txt
If I have a subdomain (a blog) that is having tags and categories indexed when they should not be, because they are creating duplicate content. Can I block them using a robots.txt file? Can I/do I need to have a separate robots file for my subdomain? If so, how would I format it? Do I need to specify that it is a subdomain robots file, or will the search engines automatically pick this up? Thanks!
Technical SEO | | JohnECF0