Why would our server return a 301 status code when Googlebot visits from one IP, but a 200 from a different IP?
-
I have begun a daily process of analyzing a site's Web server log files and have noticed something that seems odd. There are several IP addresses from which Googlebot crawls that our server returns a 301 status code for every request, consistently, day after day. In nearly all cases, these are not URLs that should 301. When Googlebot visits from other IP addresses, the exact same pages are returned with a 200 status code.
Is this normal? If so, why? If not, why not?
I am concerned that our server returning an inaccurate status code is interfering with the site being effectively crawled as quickly and as often as it might be if this weren't happening.
Thanks guys!
-
Howdie,
Yes, I believe we got this sorted out. Interestingly, it wasn't any of the suggestions made here causing the 301 status code responses. I posted a thread in Google Webmaster Tools Forum regarding the issue and received a response that I am 99.5% sure is the correct answer.
Here is a link to that thread for future readers' reference: https://productforums.google.com/forum/#!mydiscussions/webmasters/zOCDAVudxNo
I believe the underlying issue has to do with incorrect handling of a redirect for this domain: ccisound.com
I am currently pursuing getting it corrected with our IT Director. Once the remedy is in place, I should know right away if it solves the issue I am seeing in the server logs. I'll post back here once I am 100% certain that was the issue.
Thanks all! This has been an interesting one for me!
-
Hi Dana, have you definitively sorted this out?
-
They are pretty detailed, I'll send you yesterday's in a zip file so you can take a look. I'm certain that have everything needed. Thanks Eric!
-
Right, a DNS manager could do a redirect, but that would not be visible in the web server log. It would only be visible in whatever is managing the DNS.
-
Depends what kind of DNS manager you are using. A redirect via DNS can still be possible.
In my experience DNS managing software can redirect users with 301 or 302 headers depending on what settings you have. If your DNS manager has a security protocol along with redirect rules, it could be causing the issue.
Examples of DNS redirects:
-
The request headers will also show if any and what cookies the user may have set. Which it looks like is how your server determines if it should provide the client the desktop or mobile version.
-
How detailed are your log files? Can you see the user-agent (browser name) Maybe you could ask your IT department to log request headers? If that will make the log files too big, they can probably do it only for the 'problem' IPs, or only for cases that the webserver returns a 301. I'll take a look if you like. Email is in my profile.
Best,
-Eric
-
Thanks so much Eric. Yes, I was thinking about the mobile version of our site being related to what I'm seeing too. However, I am unaware that we 301 redirect anything from the main site to the mobile site. In fact, users can actually switch to the mobile site via desktop by clicking "Mobile Site" in the footer and then browse the mobile version of the site via desktop. All of the URLs are identical.
Just out of curiosity I browsed to the mobile version of our site, grabbed a URL and then plugged it into "Fetch as Googlebot" in GWT. For all options, including desktop and the three mobile options a status code of 200 was returned.
-
The problem can't be related to DNS. If the problem was related to DNS, the request would never make it to your server, and you would never see anything related to the request in your log files.
Because you can see it in your log file, it is definitely happening on your own webserver (not some external problem).
The requesting IP is probobly not the problem, but it could be if your server automatically adds to a banned list any IP that requests > X pages in Y time - your server might think this is a DOS (denial of service) attack.... But if your server was set up to do this, your IT guys would probobly know about it. This isn't something that is normally enabled 'out of the box' someone would need to intentionally activate a behavior like that.
More likely, is that there is another common denominator besides the requester IP... I would guess that it's the user agent string (the browser or device the user is using).
Taking a quick look at what I think is your site, you have a mobile version available. Google of course would be interested in what your site looks like to a mobile browser, and would send a 'fake' user agent string pretending to be so (a cell phone or a tablet etc...) If your server sees this request, and tries to automatically redirect the browser to the mobile version of the site, then you would have your 301 code (which in this case is exactly what you intended, so your all set!)
There are probably a few other cases that could cause a 301 for just some IPs, but this is the only one that comes to mind at the moment.
Good Luck!
-
Here is the response from my IT Director regarding the possibility that this is being done by our DNS manager:
"I do not believe so. Our DNS does translation of human readable names to IP address. It has nothing to do with the status being returned to a browser, and even if it did it could not write to the log file."
Is this accurate? I understand that the DNS cannot write to the log file, but if the DNS can flag a request to receive a certain status code from the server, then this scenario would still be a possibility.
-
According to our IT Director we have no spam filters, no mod_security module, absolutely nothing on our server to prevent it from being crawled by bot, human or spider from any IP address, including black-listed IPs.
To me, other than the obvious (no security is probably not a good idea at all), that means that the 301 status codes being returned because of a problem with server set up.
I do have server logs that I'd be willing to share privately with anyone who's willing to take a gander. Don't worry, I won't send you a month's worth. 1-2 days should be plenty.
In the meantime I am going to dive in and take a look further. It's entirely possible that IPs from Google are not the only ones receiving nothing but 301 status codes in response to requests.
-
Thanks William. Good suggestion. I am on it! I'll post back here once I know more.
-
I would not be surprised if this was done by your DNS. If you use a DNS manager, they could possibly redirect certain users or IPs based on patterns of visits.
I suggest finding out more about any server configurations from the admin and seeing who they use as a DNS provider or manager.
-
Excellent thoughts! Yes, they are consistently the same IP addresses every time. There are several producing the same phenomenon, so I looked at this one 66.249.79.174
According to what I can find online this is definitely Google and the data center is located in Mountain View, California. We are a USA company, so it seems unlikely that it is a country issue. It could be that this IP (and the others like it) are inadvertently being blocked by a spam filter.
It doesn't matter the day or time, every time Googlebot attempts to crawl from this IP address our server returns 301 status codes for every request, with no exceptions.
I am thinking I need to request a list of IP addresses being blocked by the server's spam filter. I am not a server administrator...would this be something reasonable for me to ask the people who set it up?
Is returning a 301 status code the best scenario for handling a bot attempting to disguise itself as googlebot? I would think setting the server up to respond with a 304 would be better? (Sorry, that's kind of a follow-up "side" question)
Let me know your thoughts and I'm going to go see if I can find out more about the spam filter.
-
Where are the 301s taking Googlebot on those IP addresses? And are they the same IP addresses every time? Have you narrowed those IP addresses down to any particular datacenter/country? It could be possible there is some configuration with your server that treats IP addresses differently depending on the country... it could also be that the IP addresses getting the 301s are known blacklisted spam IP addresses but are masking themselves as Googlebot so your server's blacklist software is keeping them out. It's really hard to say without looking into the data myself but I'm definitely interested in what you find out.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I have multiple websites for my different brands or one main website with different tabs/areas?
My client creates apps. As well as the apps they create themselves, they have made some of their own that cover various different topics. Currently they have individual websites for each of these apps, and a website for their app making business. They are asking whether they should just have one website - their app building site, which also includes information about the two apps they've built themselves. My feeling is it's better to keep them separate. The app building site is trying to appeal to a B2B audience and gain business to build new apps. AppA is trying to help carehomes and carers to streamline their business, and AppB is trying to help workplace and employee welfare. Combining them all will mean lots of mixed messaging/keywords even if we have dedicated areas on the site. I also think it will limit how much content we can create on each without being completely overwhelming for the user. If we keep them all separate then we can have a very clear user journey. I would of course recommend having blog posts or some sort of landing page to link to AppA and AppB's websites. Thoughts? Thank you!
Intermediate & Advanced SEO | | WhitewallGlasgow0 -
Tricky 301 question
A friend has relaunched a website but his web guys (he didn't consult me!) didn't do any 301s and now traffic unsurprisingly has tanked. The old site and database no longer exists and there are now 2000+ 404's. Any ideas how to do the 301s from old urls to new product urls WITHOUT it being a massive manual job?
Intermediate & Advanced SEO | | AndyMacLean0 -
Pages that 301 redirect to a 404
We are going through a website redesign that involves changing URL's for the pages on our site. Currently all our pages are in the format domain.com/example.html and we are moving to stip off the .html file extension so it would just be domain.com/example We have thousands of pages as the site deals with news so building a redirect for each individual page isn't really feasible. My plan is to have a generic rewrite rule that redirects any page that ends .html to the stripped off version of this. A problem I can see with this is that it will also redirect pages that don't exist. So for example, domain.com/non-existant-page.html would 301 to domain.com/non-existant-page which would then return a 404 status. What would the SEO repercussions be for this? Obviously if a page doesn't exist already then it shouldn't show up in the search engine indexes and shouldn't be a problem but I'm a bit worried about how old pages that currently legitimately 404 will be treated when they start to 301 redirect to a 404 instead. Not sure if there any other potential issues from this that I've missed either? Thanks!
Intermediate & Advanced SEO | | sbb0240 -
Merging 4 websites into one for a new site release (301 question)
Hi guys and girls, I have a client that has 4 very outdated websites with about 50 pages on each. They are made up like: 1 brand group and 3 for each individual key service they offer, so let's call them: brand.com (A) brand-service-1.com (B) brand-service-2.com (C) brand-service-3.com (D) We've rebuilt the main site and aggregated all the content from the others (99% re-written). Am I correct in thinking the process for the new lauch would be: 1. Launch the new site on brand.com (A) and 301 all the old brand.com (A) pages to the related pages on the new site. 2. Redirect the other websites (B,C,D) on a domain level to the new site on the brand.com (A) domain. 3. Clean up the old URL's, sitemaps, errors in Google WMT Is this right? Anything I missed/better practices? I was also wondering if I should redirect B,C,D in stages, or use page level redirects.
Intermediate & Advanced SEO | | shloy23-2945840 -
Different rankings for same keyword in different geo locations
I am listed at # 11 in google.com.pk for a same keyword but listed 4th when I use &gl=us in the same query or use a US ip address. Could this be because google's recent update and maybe they are taking time to push the update towards other countries or something else? In any case I see decent rankings when I use the query in google.com or when I use gl=us, but lower quality results in google.com.pk so please guide what is going on. Thanks in advance for guidance (y)
Intermediate & Advanced SEO | | hpk0 -
Is a 301 to a 301 ok?
I have a site that has a lot of url differences. Due to coding we sometimes have to 301 to a page that is 301'd to another. Is there any danger in doing this?
Intermediate & Advanced SEO | | EcommerceSite0 -
International SEO and server hosting
I'd appreciate feedback on a situation. We're going through a major overhaul in how we globally manage our websites. Regional servers were part of our original plan (one in Chicago, UK, and APAC) but we've identified a number of issues with this approach. Although it's considered a best practice among many, the challenges we'd face doing it are considerable (added complexity, added steps and delays to updating sites, among others). So, we shifted our plan and how are looking at hosting here in the US but to use Akami to deliver images and other heavier data pieces from their local servers (in the UK, etc.). This is how many of the larger companies like Amazon, etc. delivery their global websites. We hope that using Akami will allow us to have good performance while simplifying our process. Any warning signs we should be aware of? Anyone doing it this way and has a good experience/bad experience?
Intermediate & Advanced SEO | | josh-riley0 -
Merging three sites to one
Hi guys, I just wanted confirmation if this is the right way to go about doing this. I need to merge three websites and I've never done three websites in to a brand new site before. Ok so we have Sitex.com
Intermediate & Advanced SEO | | Profero
Sitey.com
Sitez.com We've created a SiteB.com SiteB.com has SiteB.com/SiteXCat
SiteB.com/SiteYCat
SiteB.com/SiteZCat Each X,Y and Z have over 1,000 pages. They only have about 10 pages each with Page Authority above 10 and the domains arn't that strong. What i plan to do is: 301 redirect each site domain (X,Y,,Z) to it's corresponding category. e.g. Sitex.com > SiteB.com/SiteXCat 301 redirect each page off X,Y,Z that has a Page Authority above 10 to their new pages on SiteB.com Then, I'm unsure if i should 410 every other URL... I don't think its worht 301 every single URL if they arn't in search results much - but maybe it is if they have a lot of inbound links even with low page authority? Any ideas and does the above seem the best practise? Thanks.0