Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What are the best link-building strategies for 2024, considering Google's evolving algorithms?
Link-building remains a crucial part of SEO, but with Google's algorithms constantly evolving, many traditional techniques are becoming less effective. I'm interested in learning about the latest link-building strategies that are expected to be successful in 2024. What approaches or tools should we focus on to ensure our efforts align with Google's current best practices and avoid potential penalties?
Link Building | | Ibrahim-Khuzam0 -
Backlinks on Moz not on Google Search Console
Moz is showing thousands of backlinks to my site that are not showing up on Google Search Console - which is good because those links were created by some spammer in Pakistan somewhere. I haven't yet submitted a disavow report to Google of well over 10K links because the list keeps growing every day with new backlinks that have been rerouted to a 404 page. I have asked Google to clarify and they put my question on their forum for an answer, which I'm still waiting for - so I thought I'd try my luck here. My question... If Moz does not match Google Search Console, and backlinks are important to results, how valid is the ranking that Moz creates to let me know how I'm doing in this competition and if I'm improving or not. If the goal is to get Google to pay attention and I use Moz to help me figure out how to do this, how can I do that if the backlink information isn't the same - by literally over 10 000 backlinks created by some spammer doing odd things... They've included the url from their deleted profile on my site with 100s of other urls, including Moz.com and are posting them everywhere with their preferred anchor text. Moz ranking considers the thousands of spam backlinks I can't get rid of and Google ignores them or disavows them. So isn't the rankings, data, and graphs apples and bananas? How can I know what my site's strength really is and if I'm improving or not if the data doesn't match? Complete SEO Novice Shannon Peel
Link Building | | MarketAPeel
Brand Storyteller
MarketAPeel0 -
Does Google Groups and Google Community links count?
Hello Everyone. I noticed when I added a few links from Google Groups and Google Communities. It has a do follow? Does that mean it help your SERPS? Thanks for you help?
Link Building | | EVERWORLD.ENTERTAIMENT0 -
Googles stance on Back Links via a Badge/Form
Hey guys, Does anyone know what Google's stance is on backlinks that come via a form, WordPress theme or badge. For example if I offer website security and provide badges for websites that are malware clean (with a back link to my website) and 100 websites sign up to my website will this be deemed as bad practice in Google's eyes? Also if I create a free WordPress theme with a backlink to myself? The second question sounds like I'm providing content for a link which seems okay but the first one can go either way. Thanks
Link Building | | conversiontactics0 -
Does Google index a content in iframe?
Hello guys, I´ve done a infografic and i want to share it with some bloggers. But, i want to share an iframe, not an image (beacuse there is some links on it). My question: Can Google consider the links inside an iframe (for link building)? Thank you 🙂
Link Building | | seomobly0 -
Should I 301 Redirect The index.html Page?
Hello, As an experiment I am trying to work with an SBI site but am noticing index.html is being forced on me as the homepage. As I understand it's good practice to 301 redirect the index.html page to the to the actual home page. Can someone please confirm that I am right in thinking this? Here is an example: http://www.anguilla-beaches.com/index.html
Link Building | | umtmedia
http://www.anguilla-beaches.com/ According to yahoo 773 links are pointing to the index.html page and 771 links are pointing to the / page. I'm sure Google knows that these are the same pages and does not consider them as duplicate but surely the index.html page should be redirected right? am I going mad here? Now if we look at this site: http://www.jugglenow.com the homepage has 198 links according to yahoo
http://www.jugglenow.com/index.html this page also has 198 links? now I am confused, are these 2 the same page? On this page: http://www.jugglenow.com/juggling-balls.html notice how there is no link to the actual homepage but there is a link to the index.html page. Am I missing something here because I can't figure it out, surely there should be just 1 homepage without a index.html page. Any help would be appreciated0 -
Site stuck on 2nd page on Google - how can it improve?
Hi I am working on a site http://www.harrisbassett.co.uk that for search terms such as 'Chartered Accountants Swansea' appears near the top of the 2nd page of Google. When looking at what is listed on the 1st page for 'Chartered Accountants Swansea' I would appreciate any advice/guidance on what else could be done to give the site a final push to get on the 1st page of search results. Over the past 2 weeks I have been building inbound links from local citation sites yelp, qype etc along with sites from competitor analysis via opensiteexplorer. Looking at the sites that are on the 1st page I cant understand why many of them are ranking higher than me? Any insight and plan of attack would be most appreciated from any search experts on the forum. Many Thanks
Link Building | | idv0 -
Are backlinks to a domain that redirects useless?
I am working on trying to get backlinks. We are currently redirecting a few Domains to our current domain name. One of the Domains that is redirecting is the main Domain, we will be advertising in the near future. Can I send backlinks to the redirected URL or does it only carry link juice if it is to the main Domain?
Link Building | | hfranz0