Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What are the best link-building strategies for 2024, considering Google's evolving algorithms?
Link-building remains a crucial part of SEO, but with Google's algorithms constantly evolving, many traditional techniques are becoming less effective. I'm interested in learning about the latest link-building strategies that are expected to be successful in 2024. What approaches or tools should we focus on to ensure our efforts align with Google's current best practices and avoid potential penalties?
Link Building | | Ibrahim-Khuzam0 -
slug Link redirect to subdomain?
Hi !
Link Building | | Leviiii
Im Levi new here and new in the world of SEO, please dont judge if my questions are silly. Back on the days when the site was built we thought it is a good ideea to have subdomains that together with the domain name represent our main keywords.
ex. https://stansted.tonorwich.uk, https://heathrow.tonorwich.uk, https://luton.tonorwich.uk, https://gatwick.tonorwich.uk. There is content on this subdomains, would it make any difference from SEO perspective if we create slugs that redirect to these subdomains? for example creating https://tonorwich.uk/taxi-minibus-vip-tesla-norwich-to-stansted that redirects to https://stansted.tonorwich.uk ? Or better create these slugs with slightly different content?
Any ideeas would be appreciated.
Thanks in advance!0 -
I want to know about the Impact of Do-follow backlinks and No-follow backlinks for increasing DA.
Have Do-follow backlinks and No-Follow backlinks same value to increase Domain Authority in Moz? Recently, I have created 290 profile backlinks which was do-follow. But from yesterday I have seen all backlinks are no-follow now according to the decision of sites' owners. My site's DA has already increased up to 35. Will this DA fall gradually from now? Apart from it, if I have created same backlinks for my another site; will my DA be 35? Please try to clarify it. My site is: Homeworkpaper.net Thanks. Waiting for your valuable answer.
Link Building | | darrellpc1 -
How to create link from google redirect?
I have seen redirect links from google but not sure how to create one. Please guide if anyone knows the answer. Example: https://images.google.cv/url?q=https://moz.com
Link Building | | Melissacarter3 -
Redirecting dofollow, high-domain-authority links from one site to another: good idea?
I have two sites that offer help for freelance writers. The first gets almost no search traffic and is very low priority for me. The second is my main priority and gets substantial search traffic. The low-priority site has a significant number of dofollow backlinks from high-domain-authority sites. Is it a good idea to 301 redirect these links to similar/related posts on my high-priority site? Would this potentially boost the SEO of my high-priority site? Thanks for any help!
Link Building | | John88990 -
Does an external link to an image file on my site help my SEO?
I found a few websites that are displaying images from my site. Curiously, they have also linked these images to the source image file on my website. I found these because Moz showed these sites in my list of backlinks. I assume that means it impacts my Domain Authority. But does this actually help me in search engines since it's not linking to a page on my site?
Link Building | | AlexLenhoff0 -
Spammy links redirecting to homepage of site -
Hey guys, I'm auditing a website at the moment and Moz is showing a few spammy links linking to it. The problem is these domains are all 301 redirecting to the domain I'm auditing. Let's say I'm auditing Site A. Site B is been flagged in Moz OSE with 12 spam flags. When I try to open Site B in a browser, it redirects back to Site A. What's the best thing to do, here? Disavowing domains redirecting to the site is surely a bad idea? Cheers.
Link Building | | BrianYork-AIM0 -
Need ideas to get do follow links from highly competative niche "wedding"
I need to get high quality do-follow links and my target market/keyword is ‘Maui wedding’ and a few others. Because it is such a competitive market I am having a hard time getting links from local companies providing Maui Weddings (I know why because they are my competitors) and national companies sites aren't much help because they are trying to provide leads to their advertisers. I would like do-follow links from non-paid sources any ideas or resources you could help with would be much appreciated. Thanks.
Link Building | | photoseo10