undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Intermediate & Advanced SEO
  4. Why would our server return a 301 status code when Googlebot visits from one IP, but a 200 from a different IP?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Why would our server return a 301 status code when Googlebot visits from one IP, but a 200 from a different IP?

Intermediate & Advanced SEO
5
16
3.3k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • danatanseo
    danatanseo last edited by Jun 4, 2014, 1:51 PM

    I have begun a daily process of analyzing a site's Web server log files and have noticed something that seems odd. There are several IP addresses from which Googlebot crawls that our server returns a 301 status code for every request, consistently, day after day. In nearly all cases, these are not URLs that should 301. When Googlebot visits from other IP addresses, the exact same pages are returned with a 200 status code.

    Is this normal? If so, why? If not, why not?

    I am concerned that our server returning an inaccurate status code is interfering with the site being effectively crawled as quickly and as often as it might be if this weren't happening.

    Thanks guys!

    1 Reply Last reply Reply Quote 0
    • danatanseo
      danatanseo @Christy-Correll last edited by Jun 23, 2014, 7:19 PM Jun 23, 2014, 7:19 PM

      Howdie,

      Yes, I believe we got this sorted out. Interestingly, it wasn't any of the suggestions made here causing the 301 status code responses. I posted a thread in Google Webmaster Tools Forum regarding the issue and received a response that I am 99.5% sure is the correct answer.

      Here is a link to that thread for future readers' reference: https://productforums.google.com/forum/#!mydiscussions/webmasters/zOCDAVudxNo

      I believe the underlying issue has to do with incorrect handling of a redirect for this domain:  ccisound.com

      I am currently pursuing getting it corrected with our IT Director. Once the remedy is in place, I should know right away if it solves the issue I am seeing in the server logs. I'll post back here once I am 100% certain that was the issue.

      Thanks all! This has been an interesting one for me!

      1 Reply Last reply Reply Quote 1
      • Christy-Correll
        Christy-Correll Staff last edited by Jun 17, 2014, 6:04 PM Jun 17, 2014, 6:04 PM

        Hi Dana, have you definitively sorted this out?

        danatanseo 1 Reply Last reply Jun 23, 2014, 7:19 PM Reply Quote 0
        • topic:timeago_earlier,12 days
        • danatanseo
          danatanseo @ForForce last edited by Jun 5, 2014, 6:00 PM Jun 5, 2014, 6:00 PM

          They are pretty detailed, I'll send you yesterday's in a zip file so you can take a look. I'm certain that have everything needed. Thanks Eric!

          1 Reply Last reply Reply Quote 0
          • ForForce
            ForForce @William.Lau last edited by Jun 5, 2014, 1:44 PM Jun 5, 2014, 1:44 PM

            Right, a DNS manager could do a redirect, but that would not be visible in the web server log.  It would only be visible in whatever is managing the DNS.

            1 Reply Last reply Reply Quote 0
            • William.Lau
              William.Lau @danatanseo last edited by Jun 5, 2014, 1:23 PM Jun 5, 2014, 1:23 PM

              Depends what kind of DNS manager you are using. A redirect via DNS can still be possible.

              In my experience DNS managing software can redirect users with 301 or 302 headers depending on what settings you have. If your DNS manager has a security protocol along with redirect rules, it could be causing the issue.

              Examples of DNS redirects:

              https://dnsimple.com/url-forwarding-301-redirect

              https://support.cloudflare.com/hc/en-us/articles/200172286-How-do-I-do-url-forwarding-with-CloudFlare-

              ForForce 1 Reply Last reply Jun 5, 2014, 1:44 PM Reply Quote 0
              • ForForce
                ForForce last edited by Jun 5, 2014, 12:53 PM Jun 5, 2014, 12:53 PM

                The request headers will also show if any and what cookies the user may have set.  Which it looks like is how your server determines if it should provide the client the desktop or mobile version.

                1 Reply Last reply Reply Quote 1
                • ForForce
                  ForForce last edited by Jun 5, 2014, 12:52 PM Jun 5, 2014, 12:52 PM

                  How detailed are your log files?  Can you see the user-agent (browser name) Maybe you could ask your IT department to log request headers?  If that will make the log files too big, they can probably do it only for the 'problem' IPs, or only for cases that the webserver returns a 301.  I'll take a look if you like.  Email is in my profile.

                  Best,

                  -Eric

                  danatanseo 1 Reply Last reply Jun 5, 2014, 6:00 PM Reply Quote 1
                  • danatanseo
                    danatanseo @ForForce last edited by Jun 5, 2014, 12:36 PM Jun 5, 2014, 12:36 PM

                    Thanks so much Eric. Yes, I was thinking about the mobile version of our site being related to what I'm seeing too. However, I am unaware that we 301 redirect anything from the main site to the mobile site. In fact, users can actually switch to the mobile site via desktop by clicking "Mobile Site" in the footer and then browse the mobile version of the site via desktop. All of the URLs are identical.

                    Just out of curiosity I browsed to the mobile version of our site, grabbed a URL and then plugged it into "Fetch as Googlebot" in GWT. For all options, including desktop and the three mobile options a status code of 200 was returned.

                    1 Reply Last reply Reply Quote 0
                    • ForForce
                      ForForce last edited by Jun 4, 2014, 8:58 PM Jun 4, 2014, 8:56 PM

                      The problem can't be related to DNS.  If the problem was related to DNS, the request would never make it to your server, and you would never see anything related to the request in your log files.

                      Because you can see it in your log file, it is definitely happening on your own webserver (not some external problem).

                      The requesting IP is probobly not the problem, but it could be if your server automatically adds to a banned list any IP that requests > X pages in Y time - your server might think this is a DOS (denial of service) attack.... But if your server was set up to do this, your IT guys would probobly know about it.  This isn't something that is normally enabled 'out of the box' someone would need to intentionally activate a behavior like that.

                      More likely, is that there is another common denominator besides the requester IP...  I would guess that it's the user agent string (the browser or device the user is using).

                      Taking a quick look at what I think is your site, you have a mobile version available.  Google of course would be interested in what your site looks like to a mobile browser, and would send a 'fake' user agent string pretending to be so (a cell phone or a tablet etc...)  If your server sees this request, and tries to automatically redirect the browser to the mobile version of the site, then you would have your 301 code (which in this case is exactly what you intended, so your all set!)

                      There are probably a few other cases that could cause a 301 for just some IPs, but this is the only one that comes to mind at the moment.

                      Good Luck!

                      danatanseo 1 Reply Last reply Jun 5, 2014, 12:36 PM Reply Quote 1
                      • danatanseo
                        danatanseo @William.Lau last edited by Jun 4, 2014, 8:26 PM Jun 4, 2014, 8:26 PM

                        Here is the response from my IT Director regarding the possibility that this is being done by our DNS manager:

                        "I do not believe so. Our DNS does translation of human readable names to IP address. It has nothing to do with the status being returned to a browser, and even if it did it could not write to the log file."

                        Is this accurate? I understand that the DNS cannot write to the log file, but if the DNS can flag a request to receive a certain status code from the server, then this scenario would still be a possibility.

                        William.Lau 1 Reply Last reply Jun 5, 2014, 1:23 PM Reply Quote 0
                        • danatanseo
                          danatanseo @StreamlineMetrics last edited by Jun 4, 2014, 8:22 PM Jun 4, 2014, 8:22 PM

                          According to our IT Director we have no spam filters, no mod_security module, absolutely nothing on our server to prevent it from being crawled by bot, human or spider from any IP address, including black-listed IPs.

                          To me, other than the obvious (no security is probably not a good idea at all), that means that the 301 status codes being returned because of a problem with server set up.

                          I do have server logs that I'd be willing to share privately with anyone who's willing to take a gander. Don't worry, I won't send you a month's worth. 1-2 days should be plenty.

                          In the meantime I am going to dive in and take a look further. It's entirely possible that IPs from Google are not the only ones receiving nothing but 301 status codes in response to requests.

                          1 Reply Last reply Reply Quote 0
                          • danatanseo
                            danatanseo @William.Lau last edited by Jun 4, 2014, 7:21 PM Jun 4, 2014, 7:21 PM

                            Thanks William. Good suggestion. I am on it! I'll post back here once I know more.

                            1 Reply Last reply Reply Quote 0
                            • William.Lau
                              William.Lau @danatanseo last edited by Jun 4, 2014, 7:13 PM Jun 4, 2014, 7:13 PM

                              I would not be surprised if this was done by your DNS. If you use a DNS manager, they could possibly redirect certain users or IPs based on patterns of visits.

                              I suggest finding out more about any server configurations from the admin and seeing who they use as a DNS provider or manager.

                              danatanseo 2 Replies Last reply Jun 4, 2014, 8:26 PM Reply Quote 1
                              • danatanseo
                                danatanseo @StreamlineMetrics last edited by Jun 4, 2014, 4:55 PM Jun 4, 2014, 4:55 PM

                                Excellent thoughts!  Yes, they are consistently the same IP addresses every time. There are several producing the same phenomenon, so I looked at this one 66.249.79.174

                                According to what I can find online this is definitely Google and the data center is located in Mountain View, California. We are a USA company, so it seems unlikely that it is a country issue. It could be that this IP (and the others like it) are inadvertently being blocked by a spam filter.

                                It doesn't matter the day or time, every time Googlebot attempts to crawl from this IP address our server returns 301 status codes for every request, with no exceptions.

                                I am thinking I need to request a list of IP addresses being blocked by the server's spam filter. I am not a server administrator...would this be something reasonable for me to ask the people who set it up?

                                Is returning a 301 status code the best scenario for handling a bot attempting to disguise itself as googlebot? I would think setting the server up to respond with a 304 would be better? (Sorry, that's kind of a follow-up "side" question)

                                Let me know your thoughts and I'm going to go see if I can find out more about the spam filter.

                                William.Lau 1 Reply Last reply Jun 4, 2014, 7:13 PM Reply Quote 0
                                • StreamlineMetrics
                                  StreamlineMetrics last edited by Jun 4, 2014, 4:14 PM Jun 4, 2014, 4:13 PM

                                  Where are the 301s taking Googlebot on those IP addresses? And are they the same IP addresses every time? Have you narrowed those IP addresses down to any particular datacenter/country? It could be possible there is some configuration with your server that treats IP addresses differently depending on the country... it could also be that the IP addresses getting the 301s are known blacklisted spam IP addresses but are masking themselves as Googlebot so your server's blacklist software is keeping them out. It's really hard to say without looking into the data myself but I'm definitely interested in what you find out.

                                  danatanseo 2 Replies Last reply Jun 4, 2014, 8:22 PM Reply Quote 1
                                  • 1 / 1
                                  1 out of 16
                                  • First post
                                    1/16
                                    Last post

                                  Got a burning SEO question?

                                  Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                                  Start my free trial


                                  Browse Questions

                                  Explore more categories

                                  • Moz Tools

                                    Chat with the community about the Moz tools.

                                  • SEO Tactics

                                    Discuss the SEO process with fellow marketers

                                  • Community

                                    Discuss industry events, jobs, and news!

                                  • Digital Marketing

                                    Chat about tactics outside of SEO

                                  • Research & Trends

                                    Dive into research and trends in the search industry.

                                  • Support

                                    Connect on product support and feature requests.

                                  • See all categories

                                  Related Questions

                                  • cuarto715

                                    Google Is Indexing my 301 Redirects to Other sites

                                    Long story but now i have a few links from my site 301 redirecting to youtube videos or eCommerce stores. They carry a considerable amount of traffic that i benefit from so i can't take them down, and that traffic is people from other websites, so basically i have backlinks from places that i don't own, to my redirect urls (Ex. http://example.com/redirect) My problem is that google is indexing them and doesn't let them go, i have tried blocking that url from robots.txt but google is still indexing it uncrawled, i have also tried allowing google to crawl it and adding noindex from robots.txt, i have tried removing it from GWT but it pops back again after a few days. Any ideas? Thanks!

                                    Intermediate & Advanced SEO | Mar 19, 2018, 5:56 PM | cuarto715
                                    0
                                  • Angelos_Savvaidis

                                    Multilingual Site and 301 redirection

                                    Hey there awesome people of Moz I have this site that has many languages in it. The main language is English and my developer did the following www.example.com ( is the main site ) which redirects with a 301 to www.example.com/en if your geo location is supported by our languages then you will automatically be redirected to whatever language you have in your country but does the first language with is english have to 301 redirect to www.example.com/en ? I thought that the right way is to just leave /en at the root file. Thanks in advance

                                    Intermediate & Advanced SEO | Apr 2, 2015, 12:15 AM | Angelos_Savvaidis
                                    0
                                  • ocelot

                                    Php 301 redirect

                                    Hi I am migrating an old wordpress site to a custom PHP site and the URL profiles will be different, so want to retain all link profiles and more importantly if a user visits the old urls via search then they are seamlessly transferred to the new equivalent page For example www.domain.com/about-us is going to need to redirect to www.domain.com/aboutus.php www.domain.com/furniture is going to need to redirect to www.domain.com/furniture-collections.php etc What is the best way of achieving this apart from .htaccess as not 100% confident of doing this.  Could it be done via PHP or using meta tags?

                                    Intermediate & Advanced SEO | Jul 12, 2019, 11:17 AM | ocelot
                                    0
                                  • SCW

                                    Redirect at Registrar or Server

                                    Hi folks, I have run into a situation were a new client has 3 TLDs (e.g. mycompany.com, mycompany.org and mycompany.biz), all with the same content. They are on a Windows IIS environment, which I am not familiar with. Until now, all of my clients have been Linux/Apache environment, so I always dealt with these issues utilizing htaccess. Currently all resolve to the same IP, but the URL remains the same in the browser address field (e.g. if you type-in mycompany.org - it remains as such). We want the .org and .biz version to 301 Redirect to the .com TLD. I am wondering what the best practice might be in this situation? Could we simply redirect at the registrar level or would implementation at the server level be best? If so, I would really appreciate an example from someone with experience implementing redirects on IIS. Thank you!

                                    Intermediate & Advanced SEO | Dec 19, 2013, 1:00 PM | SCW
                                    0
                                  • bloomnation

                                    DNS or 301 Website Redirect

                                    We are running a marketplace site, so we have thousands of vendors selling their products on our site. Each vendor has a Profile page and we are soon to launch a premium store-front that is white label. Many of these vendors will want to point a custom url to their premium store-front (which is a sub domain of the marketplace) and we are trying to get an understanding of how we should instruct them to point their url in a way that will give the main marketplace site the seo juice. We also want to understand what will show up in the address bar.  Will it be their url or our sub domain? Will any of the marketplace seo juice boost their url local listing status?

                                    Intermediate & Advanced SEO | Jan 3, 2013, 1:18 AM | bloomnation
                                    0
                                  • Cornel_Ilea

                                    .com and .edu difference

                                    Hello, Can anyone tell me how big is the difference between a PR5 .com and a PR5 .edu Double, triple? How big? Cornel

                                    Intermediate & Advanced SEO | Jun 13, 2012, 10:13 AM | Cornel_Ilea
                                    0
                                  • Prime85

                                    301 Redirect - What happens to backlinks

                                    Hello... One of my sites is losing rankings in G. I received the webmaster notification of unnatural links... My question is, should i do a 301 redirect of every page on my site to a new domain? If so, do the backlinks (which i believe are causing my rankings to drop) carry over? How about the good backlinks? Also, what would happen to the rankings i currently have on page 1? Thanks

                                    Intermediate & Advanced SEO | Nov 6, 2017, 1:07 PM | Prime85
                                    0
                                  • kurus

                                    Googlebot HTTP 204 Status Code Handling?

                                    If a user runs a search that returns no results, and the server returns a 204 (No Content), will Googlebot treat that as the rough equivalent of a 404 or a noindex? If not, then it seems one would want to noindex the page to avoid low quality penalties, but that might require more back and forth with the server, which isn't ideal. Kurus

                                    Intermediate & Advanced SEO | Mar 17, 2011, 3:19 PM | kurus
                                    0

                                  Get started with Moz Pro!

                                  Unlock the power of advanced SEO tools and data-driven insights.

                                  Start my free trial
                                  Products
                                  • Moz Pro
                                  • Moz Local
                                  • Moz API
                                  • Moz Data
                                  • STAT
                                  • Product Updates
                                  Moz Solutions
                                  • SMB Solutions
                                  • Agency Solutions
                                  • Enterprise Solutions
                                  Free SEO Tools
                                  • Domain Authority Checker
                                  • Link Explorer
                                  • Keyword Explorer
                                  • Competitive Research
                                  • Brand Authority Checker
                                  • Local Citation Checker
                                  • MozBar Extension
                                  • MozCast
                                  Resources
                                  • Blog
                                  • SEO Learning Center
                                  • Help Hub
                                  • Beginner's Guide to SEO
                                  • How-to Guides
                                  • Moz Academy
                                  • API Docs
                                  About Moz
                                  • About
                                  • Team
                                  • Careers
                                  • Contact
                                  Why Moz
                                  • Case Studies
                                  • Testimonials
                                  Get Involved
                                  • Become an Affiliate
                                  • MozCon
                                  • Webinars
                                  • Practical Marketer Series
                                  • MozPod
                                  Connect with us

                                  Contact the Help team

                                  Join our newsletter
                                  Moz logo
                                  © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                                  • Accessibility
                                  • Terms of Use
                                  • Privacy

                                  Looks like your connection to Moz was lost, please wait while we try to reconnect.