Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Backlinks on Moz not on Google Search Console
Moz is showing thousands of backlinks to my site that are not showing up on Google Search Console - which is good because those links were created by some spammer in Pakistan somewhere. I haven't yet submitted a disavow report to Google of well over 10K links because the list keeps growing every day with new backlinks that have been rerouted to a 404 page. I have asked Google to clarify and they put my question on their forum for an answer, which I'm still waiting for - so I thought I'd try my luck here. My question... If Moz does not match Google Search Console, and backlinks are important to results, how valid is the ranking that Moz creates to let me know how I'm doing in this competition and if I'm improving or not. If the goal is to get Google to pay attention and I use Moz to help me figure out how to do this, how can I do that if the backlink information isn't the same - by literally over 10 000 backlinks created by some spammer doing odd things... They've included the url from their deleted profile on my site with 100s of other urls, including Moz.com and are posting them everywhere with their preferred anchor text. Moz ranking considers the thousands of spam backlinks I can't get rid of and Google ignores them or disavows them. So isn't the rankings, data, and graphs apples and bananas? How can I know what my site's strength really is and if I'm improving or not if the data doesn't match? Complete SEO Novice Shannon Peel
Link Building | | MarketAPeel
Brand Storyteller
MarketAPeel0 -
Why Google Search Console Data is different from Moz Data for my website
Hi All I am running a website, I have been using Moz since Feb 2021. Kindly go through these pics My question is why Moz is showing 2K plus backlinks while Google search console is showing just 1253 backlinks. Why fewer links in the google search console is less? How can I increase Google search console backlinks? Also, Moz is showing 90+ DA backlinks but those websites are not showing by the Google search console. What should I do to let google consider them? m0JGjBQ.jpg Cmp8ei3.png
Link Building | | ssubodhsingh0 -
Spammy no follow backlinks - what should I do?
There is a individual/company that is creating lots of spam sites in my niche. It doesn't seem that they are trying to rank for any keywords, so I'm a little confused as to their purpose, but here's what they're doing. They scrape the top 10 Google results for each keyword and create a page - so if the niche is "widgets", they scrape the top 10 ranked sites for 'widgets', 'blue widgets', 'red widgets', and so on. A 'results' page is created for each keyword, which is linked from a home page. The results page always contains 10 websites - text is the site metadescription or similar. Each website gets a nofollow link back with the page title used as the anchor text. The host sites all have the keyword in the domain name - e.g www.widgetsxyz.com. The sites are thrown together but interestingly have a very crappy but individual logo (like 90's clipart). The host sites all have high DA/PA due to some very extensive link spam pointing to the sites, with very targeted keywords. There is seemingly no purpose to these sites that I can see - no other 'followed' links on the page or site. Whoever is doing this is churning out tens, if not hundreds of these sites. Any ideas what might be going on here, and whether I should be disavowing these sites (even though no follow). The fear is that they could switch the links to do follow, and get all the major players penalised in one fell swoop. ???
Link Building | | johnohara0 -
Googles stance on Back Links via a Badge/Form
Hey guys, Does anyone know what Google's stance is on backlinks that come via a form, WordPress theme or badge. For example if I offer website security and provide badges for websites that are malware clean (with a back link to my website) and 100 websites sign up to my website will this be deemed as bad practice in Google's eyes? Also if I create a free WordPress theme with a backlink to myself? The second question sounds like I'm providing content for a link which seems okay but the first one can go either way. Thanks
Link Building | | conversiontactics0 -
Links from PRWeb press release violate Google's quality guidelines?
My site has had a manual action performed on it by Google indicating that I have inbound links that fall outside of their quality guidelines. I did my own research, found what I thought was the issue, had the links removed and requested reconsideration. Google's response surprised me in that they highlighted two specific pages with links that were the direct result of valid press releases and a publisher picking up our release off a wire service. Has anyone else seen this occur? Anyone had a case successfully reconsidered? I realize that I don't need to do anything at all as the manual action is in effect and will stay that way, discounting those links, but I would rather a) not have any manual action against my site and b) know for the future so this doesn't happen again. Also, is this applicable for guest blog posts, which effectively create the same type of backlinks? Thanks
Link Building | | barberm1 -
Do inbound links to download .exe files improve SEO?
The site, (http://www.bartenderbarcodesoftware.com/) sells barcode software. They also provide free printer drivers on their site. During a recent domain change (before I started helping them), they began looking at their inbound links and found quite a few external sites linking to these download files. However, when someone clicks on the link on one of these sites (i.e. http://vvs-motor.com.ua/barcode-printer-drivers) it immediately begins downloading the file (i.e. http://www.seagullscientific.com/downloads/drivers/archive/7.2/7.2.4/Unimark_7.2.4.exe) without taking the visitor to the site. The owner would greatly prefer this traffic to engage with the site and is considering 301 redirecting those download links to an actual webpage. Question: Would this look suspicious to Google? Would this have a positive or negative effect on SEO? The last factor to consider is that many of these sites will likely just recreate the list based on the new links at some future point. Thanks for any help!
Link Building | | motosantiago0 -
Does the ratio of external nofollow links to external "do follow" links matter in terms of SERPs ranking?
My site has an external link nofollow:dofollow ratio of approximately 1:1 That is, there are about as many nofollow external links as "do follow" external links. I have an impression that the ratio of no-follow to "do follow" links is a factor in the way that our website shows up in SERPs. I have the impression from reading a variety of sources, and from looking at Seomoz, that calculate "trust" factors as if they mattered (in SERPs), that seem to value a relatively low nofollow:dofollow ratio. Am I correct about that? Thanks,
Link Building | | tcolling
Tim PS - I don't know whether or not this matters, but our website is at: www.trustworthycare.com - Tim0 -
Value of Link Redirect from Google News
Some sites are linking to us with the URL from Google News instead of using a direct link. For example: "http://news.google.com/news/url?sa=t&fd=R&usg=ghjgjhggjhgkjghjg&url=http://www.oursite.com/awesome-news-010123/" Does this pass the same value as a normal link? I asked the publisher to replace the link with a normal link but they pushed back saying that the news URL is better.
Link Building | | ProjectLabs0