Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to Increase Website Visibility on Google and Bing?
I am Working on Tech Niche Website My Website Title is Azeem Safi: Technology News, Gadgets, Howto, Reviews & More and I want to Rank on Google and Drive more traffic on Google. Please Guide me
Link Building | | ciwili31150 -
How to create link from google redirect?
I have seen redirect links from google but not sure how to create one. Please guide if anyone knows the answer. Example: https://images.google.cv/url?q=https://moz.com
Link Building | | Melissacarter5 -
Redirects affect on page and domain authority
Hi - excuse me if this is stupid question but I'm rather new to this. Currently the PA for the home page is 16 and the DA 28 and I have some concerns that my efforts are being diluted. When I use OSE on www.sassandgrace.co.uk I get different link results to sassandgrace.co.uk but there is a redirect in place to sassandgrace.co.uk and cannot run a crawl report, for example, on www.sassandgrace.co.uk. The PA and DA are identical for each but the number of links different With a smallish number of links pointing to each but with a redirect in place, am I losing some of the effect or is having the redirect in place giving me all the credit for links to both addresses? Thanks
Link Building | | Sassandgrace0 -
How to make google index the page title has changed ?
Google has indexed all my websites, but I have to change some title in the website but when checking google does not update the change.
Link Building | | tuananh0688
Please help me .
My site : http://blog.addme.vn0 -
Could google recognize urls in data-attributes?
Hi, could google's crawler recognize the following example as a link and will be crawl the url? any experience? ty for your help
Link Building | | censeo0 -
Does the ratio of external nofollow links to external "do follow" links matter in terms of SERPs ranking?
My site has an external link nofollow:dofollow ratio of approximately 1:1 That is, there are about as many nofollow external links as "do follow" external links. I have an impression that the ratio of no-follow to "do follow" links is a factor in the way that our website shows up in SERPs. I have the impression from reading a variety of sources, and from looking at Seomoz, that calculate "trust" factors as if they mattered (in SERPs), that seem to value a relatively low nofollow:dofollow ratio. Am I correct about that? Thanks,
Link Building | | tcolling
Tim PS - I don't know whether or not this matters, but our website is at: www.trustworthycare.com - Tim0 -
How does Google treat multiple backlinks on the same page?
Hello mozCrew Just a quick yay or nay. We have an embeddable code on our site for customers who want to post our link on their website listed here: http://www.accupos.com/etc/embed-code-restaurants.php But in our link, there are actually 2 separate links, one branding link that says the name of our company, AccuPOS, and another link which targets a specific target keyword "Restaurant POS Software". My question is, does Google only give credit for the branding link, since it is listed FIRST? Or will we get double credit? Is this a good or bad practice, and what would you recommend? Thank you! Derek Moore
Link Building | | DerekM880 -
Are backlinks to a domain that redirects useless?
I am working on trying to get backlinks. We are currently redirecting a few Domains to our current domain name. One of the Domains that is redirecting is the main Domain, we will be advertising in the near future. Can I send backlinks to the redirected URL or does it only carry link juice if it is to the main Domain?
Link Building | | hfranz0