Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Slug redirect to existing subdomain?
Hi everyone, I'm Levi, new here and new in the world of SEO so please don't judge if my question is silly. Back on the days when my website was built we tought of having subdomains to help with SEO. We have a couple subdomains set up https://stansted.tonorwich.uk, https://heathrow.tonorwich.uk, https://gatwick.tonorwich.uk, https://luton.tonorwich.uk etc .
Link Building | | Leviiii
We do have some content on these, but from seo perspective creating a slug and redirecting it to the subdomains would be beneficial? Would it make any difference at all, good or bad? Ex. Creating
https://tonorwich.uk/taxi-minibus-tofrom-stansted and redirecting it to https://stansted.tonorwich.uk
Will this help or we better create different content on the slugs? Waiting for your opinions.
Thanks in advance.0 -
301 redirect from a de-indexed site
Is link juice passed from a website that has been intentionally de-indexed by webmaster? The scenario is I had 2 websites with similar content. I decided to get rid of one by de-indexing. I also did a 301redirect from my de-indexed site. Not the whole domain but just the 2 URL's which I believe could pass on link juice. In this case how long the link juice would flow via 301 redirect?
Link Building | | riyaaaz0 -
01 November 2013 - A possible Google's update??
Hello Guys! On 01 November 2013, one of our website's traffic has been dropped by more than 70%. We had added plenty of domains to the disavow tool, I am just wondering if its a Google's update or the disavow tool has destroyed the site? Any ideas? 9p3sN3p.jpg
Link Building | | TheSEOGuy10 -
Google Penalty
Hello thereIt has just been confirmed that my site is suffering for a Google Penality. The odd thing is that it is just for my set of key keywords and not the whole site. It seems in 2010 that a rouge SEO company I used sent loads of spammy links but I have lots of other good links too. What is your advice moving forward? Will it drop over time? Do I set up a new site (this has been established over 10 years) or try and get better quality links to compensate for the spam ones? Thanks so much! Kindest regards Victoria
Link Building | | vcasebourne0 -
How to build links for a domain name that is redirected to another url?
I'm helping a friend build links to his site because he recently moved across country. His primary domain isn't very SEO friendly (shef1.com) so he registered another domain (stephenshefrinphotography.com). Shef1.com has been around for awhile and has at least been crawled with a page rank of 1 so we decided to keep shef1.com as the main website and redirect stephenshefrinphotography.com to it. So for SEO sake I know it's best to at least have a keyword in your URL so I have been using the url stephenshefrinphotography.com in my link building efforts for local directories and blog comments. Is this the right way to go about this or should I be using the main url shef1.com? He had virtually no backlinks until I started helping him but the ones I have created over the last month are not showing up on the link analysis in my campaigns. I just want to make sure my efforts aren't in vain. Advice please. Thanks!
Link Building | | mrsmelmitch0 -
How to make google index the page title has changed ?
Google has indexed all my websites, but I have to change some title in the website but when checking google does not update the change.
Link Building | | tuananh0688
Please help me .
My site : http://blog.addme.vn0 -
How often does Google crawls old pages, is it worth to get links from them ?
Let's say there's a blog post in my industry niche that's 2 years old. Is it worth to get a link from that blog post ? How do I know Google will actually re-crawl that page, and how long can that process take ?
Link Building | | pqdbr0 -
No Follow Blogs. Do they help with keyword density?
I know no follow blogs don't pass page rank or link juice, but do they aid in help with making your site know for a specific keyword when you use anchor text?
Link Building | | photoseo10