Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Linking out: Follow vs. Nofollow
When linking out to other sites, such as resources on .gov pages, should I use follow or nofollow links?
Link Building | | AHC_SEO0 -
Yandex vs. Google
I own an e-commerce website and my primary keyword ranks on page one of Yandex, but page 12 of Google. I have had some (likely less than optimal) links built to my website over the years. Considering Yandex doesn't take links into account, which is more likely? a. My high ranking on Yandex is indicative of where I COULD be ranking on Google if the unnatural links weren't being held against me. b. My page 12 ranking on Google is indicative that because links are still of value in Google, all of the sites above me have more and/or better quality of them. For example, A LOT more big name sites rank highly in Google as opposed to Yandex... presumably because they have a ton of good links that I (and other smaller sites) can't naturally get. Thoughts?
Link Building | | riahc0 -
Google to devalue guest blog links?
There's a lot of talk at the minute about google possible devaluing guest post links. I wondered what people thought of this and whether it is likely to happen? If it does happen then how will google identify 'spam' guest posts and links? I guess it's fair to assume that if the post is well written, autoriative and adds value to a site with a knoweldgable perspective then it will be counted? It can be very hard to define how google will define a 'quality guest post'? Any ideas? Here's what Matt has to say... http://www.webpronews.com/matt-cutts-weighs-in-on-guest-blogging-for-links-2012-10
Link Building | | SamCUK0 -
Making sure links get indexed by google?
Hi I currently submitted to 30 seo directories & I just want to make sure my links get indexed by google so the link value passes onto my site. Any suggestions about getting my links indexed? I was thinking of either pinging or social book marketing the links! What's your view, thanks.
Link Building | | Socialdude0 -
Why Google is not getting many of the links to my site?
Hello, I have a website for cosmetics, selling and a lot of information (http://www.vipcosmetica.com). For about half a year I've been working also in seo for several google keywords. My question is about the external links to my site. Using different tools, including those of SEOMoz, I see just a few links to my site, however there are plenty, because I've been putting them myself, in public directories, related blogs, industry websites and so on. Alexa shows 5 links only, OpenSiteExplorer shows 4, and Market Samurai 9. I thinks theres should be like 50-60 links, minimum, in pages with good google PR (Youtube, Vimeo, Vulka, Hotfrog, Digg, Flickr, etc.). In Alexa, recently two more links appeared, from two blogs that have copied some content from my site (priveta policy) without changing the original links that point to other pages on my site, and those are the links that Alexa shows now. What could be the reason the rest of the links do not appear at all? Thank you very much
Link Building | | MadridOriflame0 -
Will redirecting 250 linking domains to a new website from an old website freak Google out?
Prior to my accepting a project, the client decided to change the name of their company and purchased a new domain name to match. They hired a designer to build their new company site which has now been up and running for a little under 12 months. Up to this point they have done a good job of slowly redirecting small handfuls of linking domains from the old site to the new. They want to make one last push and point the remaining 250 linking domains from their old site to their new site. Will redirecting 250 linking domains to a new website from an old website freak Google out? If so, how many domains should be redirected at one time? My main concern is not wanting to upset Google and see the floor fall out on current keyword rankings, domain authority and page rank, etc.
Link Building | | 5outhpaw0 -
How Long for Google Directory and Search.com to update with Current Dmoz.org Data?
So I recently had a site added to dmoz.org and I'm wondering if anyone knows when Google Directory & Search.com update their site with new Dmoz.org data? They both use dmoz.org for their directory listings. Regards,
Link Building | | qlkasdjfw0 -
No-Follow Tag and Advertising
Hiya! I've got a relatively new domain, and one of my goals is to be 100% white hat, squeaky clean as I really would love to see my site succeed in the long term. So far my linkbuilding efforts are going pretty well. I'm beginning to buy some ads, and some of the bloggers I'm buying ads from don't use advertising networks, they just host your jpeg from their own blog... For the little picture that will show up in the side column of their blog should this be no-followed? And if it's not do we know if google assigns a penalty? I'm ok with asking the blogger to do that, but I notice my competition is not (and I know they know better) so I feel like it puts me at a bit of a disadvantage. Secondly - I'm 100% avoiding sites that sell text links (even ones with high pagerank, like pagerank 6), is this the right path to go? I read this article from Matt Cutts and was wondering if this is still true, reputable sites that sell links maintain their pagerank but aren't able to pass pagerank? http://www.mattcutts.com/blog/text-links-and-pagerank/ "Reputable sites that sell links won’t have their search engine rankings or PageRank penalized–a search for [daily cal] would still return dailycal.org. However, link-selling sites can lose their ability to give reputation (e.g. PageRank and anchortext)."
Link Building | | super990