Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google isn't crawling my website page
I have been trying to get indexed my inner page's title tag but still I cannot see this. It has been almost a week. I have done "Fetch as Google" even though I cannot see this. What can be the reason?
Link Building | | ksmith880 -
Website not appearing anywhere on google
Hello, we launched www.comparetravel.ie in June but it is not showing up on google search, we have submitted it to google twice but still no sign of our site. I thought it would at least show up for the "Compare Travel" search term, it only shows up when you type the full website address into google. Any ideas on what we are doing wrong? Alll advice greatly appreciated. (The website is Irish based) Thanks P
Link Building | | CompareTravel0 -
Does the ratio of external nofollow links to external "do follow" links matter in terms of SERPs ranking?
My site has an external link nofollow:dofollow ratio of approximately 1:1 That is, there are about as many nofollow external links as "do follow" external links. I have an impression that the ratio of no-follow to "do follow" links is a factor in the way that our website shows up in SERPs. I have the impression from reading a variety of sources, and from looking at Seomoz, that calculate "trust" factors as if they mattered (in SERPs), that seem to value a relatively low nofollow:dofollow ratio. Am I correct about that? Thanks,
Link Building | | tcolling
Tim PS - I don't know whether or not this matters, but our website is at: www.trustworthycare.com - Tim0 -
Links or Redirect?
My client's company has just acquired another company in the same industry and with that acquisition came two websites; companyname.com and companyname.co.uk. companyname.com has PA13 and DA10, companyname.co.uk has PA15 and DA10. At the moment, the two companies are to be run as separate companies but will become fully merged in time. My client's own website is brand new (PA1 and DA) and I'm wondering what is the best way to use these other two sites to improve my client's search ranking. Should I use the two sites to link to my client (these would be logical and natural links e.g. explaining the acquisition, under new management) or would it simply be better to redirect the domains to the new site? Many thanks
Link Building | | mikecartmel0 -
What is white hat seo for google penguin and google panda?
Hi, This is my 1st question to some experts. I need advice on link building. Are following good and white hat seo? 1: Bookmarkings on top sites 2: Top directory submission, or directories that seomoz recommended? 3: Top article directories, yahoo voices, ezine etc.. 4: Top web 2.0, wordpress, blogger, tumblr 5: Guest posts on quality blogs. I was use to with black hat methods (by mistake) , that i was not aware, Some are pure blackhat, that i never did. some time it hard to judge which are black hat. In past google like some links, now most are devalue or got penalty. Now i want to do quality seo work and slow process. any advice will be great regarding quality seo. Thanks.
Link Building | | chsajid110 -
Google Penalty
Hey there, I am working with a site who recently received an email from Google that they have been penalized. Since I have only been working with this site for a couple of weeks I know it's from past SEO work. I tried to pull back links for them and they have 1.5 million of them and every one I look at is a good quality site. I did notice one thing that stood out and I need your opinion. They have a blog in which they run about 3 ad banners and about 4 other links site wide creating several thousand links with the same anchor text and seems to be a bit spammy on the page. However, the blog is owned by them... and the amount of same anchor text links is a very small percentage of their 1.5 million links. Can you offer a different way for me to attempt to find these bad links? Could their blog by itself cause the issue? Please share your thoughts with me about this.
Link Building | | PageOnePowerGang0 -
Why Google is not getting many of the links to my site?
Hello, I have a website for cosmetics, selling and a lot of information (http://www.vipcosmetica.com). For about half a year I've been working also in seo for several google keywords. My question is about the external links to my site. Using different tools, including those of SEOMoz, I see just a few links to my site, however there are plenty, because I've been putting them myself, in public directories, related blogs, industry websites and so on. Alexa shows 5 links only, OpenSiteExplorer shows 4, and Market Samurai 9. I thinks theres should be like 50-60 links, minimum, in pages with good google PR (Youtube, Vimeo, Vulka, Hotfrog, Digg, Flickr, etc.). In Alexa, recently two more links appeared, from two blogs that have copied some content from my site (priveta policy) without changing the original links that point to other pages on my site, and those are the links that Alexa shows now. What could be the reason the rest of the links do not appear at all? Thank you very much
Link Building | | MadridOriflame0 -
No-Follow links on clients blog?
I have a client who has a blog with a side bar on each page, in the sidebar there are various links to the site, as well as links to some of his partner sites. I was looking at the recently updated open site explorer, and noticing that their domain authority is way down, a good amount below their competitors as well. The total links compared to the linking root domains is ~3000 to 200. So my question is 1. How can I gain on the domain authority front? (more diverse links?) And if so, 2. Should I no-follow the links on their blog so they don't have hundreds of pages with the same links (most internal) on there site Thanks,
Link Building | | USHoleInOne
Nick0