Do I have a robots.txt problem?
-
I have the little yellow exclamation point under my robots.txt fetch as you can see here- http://imgur.com/wuWdtvO
This version shows no errors or warnings- http://imgur.com/uqbmbug
Under the tester I can currently see the latest version. This site hasn't changed URLs recently, and we haven't made any changes to the robots.txt file for two years. This problem just started in the last month. Should I worry?
-
Today it has a green check mark, and absolutely no changes were made to the website since I asked this question.
-
It could be that your server had a hard time when Google tried to view your robots.txt file that's why it wouldn't be able to fetch it. As long as this issue doesn't prevent Google anymore in the future it's not much to worry about.
-
That would make me feel more confident of a false error being reported. Time to closely monitor the crawl logs, look at server stats, and keep an eye on GWT for a change in the reporting/indexing. I would also go into the GWT forums and post, see if anyone is reporting a similar error these past couple days.
-
I can't post the domain but I know it is accessible.
When I go to the tester it shows the live robots.txt with no problems. I also can look at the server logs and see that it is being crawled, but being crawled less then Bing Crawls. Also the Bing Webmaster Tools is showing no problems.
-
Can you post your domain? Manually checking the robots.txt file would help.
I've checked many of my GWT accounts and I am not showing a sudden robots.txt error. It could be a false error, but I would take anything with the robots.txt file seriously. You'll want to make sure that it is in fact accessible to all the crawlers desired.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
No index tag robots.txt
Hi Mozzers, A client's website has a lot of internal directories defined as /node/*. I already added the rule 'Disallow: /node/*' to the robots.txt file to prevents bots from crawling these pages. However, the pages are already indexed and appear in the search results. In an article of Deepcrawl, they say you can simply add the rule 'Noindex: /node/*' to the robots.txt file, but other sources claim the only way is to add a noindex directive in the meta robots tag of every page. Can someone tell me which is the best way to prevent these pages from getting indexed? Small note: there are more than 100 pages. Thanks!
Technical SEO | | WeAreDigital_BE
Jens0 -
I hope someone can help me with page indexing problem
I have a problem with all video pages on www.tadibrothers.com.
Technical SEO | | TadiBrothers
I can not understand why google do not index all the video pages?
I never blocked them with the robots.txt file, there are no noindex/nofollow tags on the pages. The only video page that I found in search results is the main video category page: https://www.tadibrothers.com/videos and 1 video page out of 150 videos: https://www.tadibrothers.com/video/front-side-rear-view-cameras-for-backup-camera-systems I hope someone can point me to the right way0 -
Will it be possible to point diff sitemap to same robots.txt file.
Will it be possible to point diff sitemap to same robots.txt file.
Technical SEO | | nlogix
Please advice.0 -
Site redesign. Possible SEO problems?
Hi! Our website has enjoyed good rankings for lots of our keywords for the past 9years. Over the years the site became heavy, so we want to change the design radically. Content, filenames, titles, urls etc. will all remain identical just the change of a template. The new one is minimalistic and responsive design. We are worrying this may drop our domain or page authority and kill all our past SEO efforts. **How to do this with the least harm to our rankings? Anything we need to avoid?**Articles/advice/suggestions comments on design and SEO would be greatly appreciated. Thank you for your help! Alina
Technical SEO | | skalfa0 -
Importance of WMT change of address and problem doing it
Hi, How important is it to submit a change of address in WMT? I say that because I am having problems doing it so wondered if it was worth the hassle in trying to fix it. I am getting the error: "We couldn't verify website.co.uk. To submit a change of address, website.co.uk must be verified using the same method as www.website.co.uk. Add website.co.uk to your account and verify ownership, then try again." I have looked on the web to try and find an answer and have come across 2 suggestions: You might have lost the verification with the redirect. If you used a metatag on the home page, the home page is now redirecting. If you had uploaded a verification text file, that file is probably now gone and redirecting as well. You probably need to re-verify the site. Either re-upload the text file and configure it not to redirect (may be difficult) or use the DNS server verification method. You need to verify the non www. version of the website because that's the way Google like it. Not sure why solution 2 would be necessary but it does seem to be what WMT are getting. Because the site already redirects, 1 would then come in to play. Is it worth persevering with because IT will be getting a long list of stuff to do from me as it is.... Thanks all
Technical SEO | | Houses0 -
Problem www/non-www domain rewrite
Hello, I've made a site for a client about 1 year ago. The rankings are quite okay, but the home page suffers from a penalty I think. I found out via OSE that PageAuthority strangely is higher on the 301-ed page www.myanmar-rundreisen.de - PA 32
Technical SEO | | hgw57
myanmar-rundreisen.de/ - PA 33 I don't understand what is happening here as I am using the usual htaccess 301-redirect: Rewrite domain.com -> www.domain.com RewriteCond %{HTTP_HOST} .
RewriteCond %{HTTP_HOST} !^www.myanmar-rundreisen.de [NC]
RewriteRule (.*) http://www.myanmar-rundreisen.de/$1 [L,R=301] which is working fine with other domains ... I tried also (last line) RewriteRule (.*) http://www.myanmar-rundreisen.de/$1 [L,R=301] So thanks to anyone who can share an idea on that ... Guenter K04xy.jpg0 -
Problem with re-direction
Please help me to solve a problem with redirection. I re-do the
Technical SEO | | NadiaFL
site and move it to new domain page-by-page as recommended. I use WordPress Redirect plugin. I did everything as written 10
days ago but don't see redirection. For example, old page http://aurora17.com/?page_id=2485 New page http://njcruise.org/alaska-cruise-tour/ Where is a problem? How to solve it?0 -
Search engines have been blocked by robots.txt., how do I find and fix it?
My client site royaloakshomesfl.com is coming up in my dashboard as having Search engines have been blocked by robots.txt, only I have no idea where to find it and fix the problem. Please help! I do have access to webmaster tools and this site is a WP site, if that helps.
Technical SEO | | LeslieVS0