Old domain still being crawled despite 301s to new domain
-
Hi there,
We switched from the domain X.com to Y.com in late 2013 and for the most part, the transition was successful. We were able to 301 most of our content over without too much trouble.
But when when I do a site:X.com in Google, I still see about 6240 URLs of X listed. But if you click on a link, you get 301d to Y. Maybe Google has not re-crawled those X pages to know of the 301 to Y, right? The home page of X.com is shown in the site:X.com results. But if I look at the cached version, the cached description will say :This is Google's cache of Y.com. It is a snapshot of the page as it appeared on July 31, 2014."
So, Google has freshly crawled the page. It does know of the 301 to Y and is showing that page's content. But the X.com home page still shows up on site:X.com. How is the domain for X showing rather than Y when even Google's cache is showing the page content and URL for Y?
There are some other similar examples. For instance, you would see a deep URL for X, but just looking at the <title>in the SERP, you can see it has crawled the Y equivalent. Clicking on the link gives you a 301 to the Y equivalent. The cached version of the deep URL to X also shows the content of Y.</p> <p>Any suggestions on how to fix this or if it's a problem. I'm concerned that some SEO equity is still being sequestered in the old domain.</p> <p>Thanks,</p> <p>Stephen</p></title>
-
Hey Stephen,
Did you ever solve this? I'm experiencing the exact same issues you've described above, so I'm curious if you ever figured it out.
Thanks.
-
If using a PHP CMS you will have a page # eg ?P123 this will make it so you can collect all of the old pages and have them redirect to the new ones perfectly.
Turn off all catching frist
A close friend of mine and I did this on his site which is quite large he got 30,000 URLs that were not pointing from his old domain to his new domain simply by adding the code below. If running WordPress and Nginx follow the steps below if you need to modify them so they correspond with your page format use
http://danielmiessler.com/blog/redirect-archives-pages/
Check out Yoast
https://yoast.com/wp-content/permalink-helper.php
&
https://yoast.com/change-wordpress-permalink-structure/
Sorry for being MIA,
Tom
-
Hi Matt, thanks for the reply. I can give this a test.
The puzzling thing here is that Google has re-crawled those old page URLs at the old domain. It is getting 301 re-directed to the new URL. The content that Google has cached for the old URL is from the new URL and has the new URL listed as the source of the cached page.
Stephen
-
Force a recrawl by pinging the pages.
Get as complete a list as possible of the pages that are indexed. Then submit them to PingFarm, 247pinger and RankonTop.
We had 2500+ pages indexed from an old development server and this removed almost all of them in 2-3 weeks.
-
Hi David, you can see my responses to these questions in my reply to Thomas. But the URLs from old X.com are being 301d correctly to y.com. For the vast majority of duplicate listings shown for site:x.com, you can click on them and they get 301d to y.com and the cached version of a page in site:x.com shows y.com content and even the y.com URL in the cached description.
-
I am going to assume the 301 redirects are working when you click on them?
Hi Thomas,
Yes, the 301s work and confirmed with http headers
_You need to go into Google Webmaster tools and tell Google that you have moved from domain A to domain B this will make Google bot crawl both sites vigorously looking for 301 redirects. _
The migration was before my time, but I think that this was done. If I try to use the change of address on the old site, X.com, I just get a "There is no change of address pending for your site." message with no control options (conversely, if I tried to do the same thing for the current Y.com, it would give me the full list of instructions on how to migrate my site.)
I want to be certain that you did not 301 redirect domain to domain and not page to page. So if site X has homepage, about page, whatever page, and site Y would contain the exact same pages or equivalent pages that you would have already 301 redirected page 2 page not just point the domain at the other domain is that right?
The 301s are at a page level at X.com to the same equivalent page at Y.com. The majority of pages have migrated over without obvious problems. But it's a little disturbing to see new pages of y.com which have never been part of x.com somehow make it into a site:x.com query and listed with an x.com domain to start the URL.
So just to recap, site:x.com shows long-tail pages that clearly belong to site:y.com and were never part of x.com. The <title>of some of the site:x.com pages. for instance, are definitely from y.com pages. For some reason, Google is associating these pages with the x.com domain.</p> <p>If you click on the cached version of a listing for site:x.com, the cached version will show the content and URL of Y.com/foo in the cached description header. Clicking on the actual link gets you 301d from X.com/foo to Y.com/foo. Both events indicate that the 301 is working and that Google is recognizing the 301.</p> <p>I don't know if this impacting our SERPs or not. If I do a very page-specific search for "blue widgets A, B, and C in Montana" for a page that is indexed in both site:x.com and site:y.com, you only see y.com's page which is expected behavior. You don't see x.com in the SERPs for that specific query. It's only if you do site:x.com "blue widgets A, B, and C in Montana" do you see the duplicate listing. But again, clicking on the URL that is shown results in a 301 to the proper y.com page.</p> <p>I can dig deeper with my developers, check logs, etc. But it's weird. It's almost like Google sees a URL for y.com and indexes that URL for x.com and y.com even though it knows that y.com is the dominant or real page (evidenced by cached data info). Everytime I click on an site:x.com link, I get 301 redirected properly to the y.com equivalent.</p></title>
-
Did you keep the page URL structure the same on the new site? If so, you can do a simple htaccess rule that will forwards all traffic from one domain, to the corresponding pages on the other domain. If not, then you will have to go through and check each one manually.
Have all the steps been covered? Did you resubmit a new sitemap? Specify in "fetch as Google" that all the new pages on the new site be crawled? If you are using a CMS, did you check to see if multiple versions of a certain page exist, and forget to redirect those?
Lastly, even though you still see references to the old domain, do the 301 redirects work? Once clicked do they send a user to the new domain?
-
I am going to assume the 301 redirects are working when you click on them?
Going on that assumption and please correct me if I am wrong. You need to go into Google Webmaster tools and tell Google that you have moved from domain A to domain B this will make Google bot crawl both sites vigorously looking for 301 redirects. Basically any changes that have occurred when the first crawl is completed after approximately 90 days tell Google to crawl it again. You have a fairly large site based on the amount of links you are discussing. You need to get Google to find everything. That is the best way to tell Google that you are changing domains.
https://support.google.com/webmasters/answer/55281
Two references I would use. Each contains a lot of good information.
https://support.google.com/webmasters/topic/6033102?hl=en&ref_topic=6029673
&
http://moz.com/community/q/how-to-keep-old-url-juice-during-site-switch
"But when I do a site:X.com in Google, I still see about 6240 URLs of X listed. But if you click on a link, you get 301d to Y. Maybe Google has not re-crawled those X pages to know of the 301 to Y, right? The home page of X.com is shown in the site:X.com results. But if I look at the cached version, the cached description will say :This is Google's cache of Y.com. It is a snapshot of the page as it appeared on July 31, 2014."
I want to be certain that you did not 301 redirect domain to domain and not page to page. So if site X has homepage, about page, whatever page, and site Y would contain the exact same pages or equivalent pages that you would have already 301 redirected page 2 page not just point the domain at the other domain is that right?
I hope this helps,
Thomas
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Value of domain name for domain authority. Please help to figure out!
I am doing SEO for an appliance repair company. Their company website's domain doesn't have high authority, and I am going to increase that by link earning and content improving. I think a better domain name might also help me out. The current URL contain the word "appliance" but doesn't have "repair" in it. I am thinking a new domain that would contain both keywords will serve better. Could you please share with me your thought on this? Am I in the right direction, or not at all? I know Google penalizes mirror sites since this they are considered as duplicated content. I'll upload my content to the new domain and make the old one point to that new URL. I am wondering if canonical might help? Or 301 redirect will be a better solution? Any advise would be highly appreciated! Thank you!
Technical SEO | | kirupa0 -
Clients domain expired - rankings lost - repurchased domain - what next?
Its only been 10 days and i have repurchased the domain name/ renewed. The who is info, website and contact information is all still the same. However we have lost all rankings and i am hoping that our top rankings come back. Does anyone have experience with such a crappy situation?
Technical SEO | | waqid0 -
Correct linking to the /index of a site and subfolders: what's the best practice? link to: domain.com/ or domain.com/index.html ?
Dear all, starting with my .htaccess file: RewriteEngine On
Technical SEO | | inlinear
RewriteCond %{HTTP_HOST} ^www.inlinear.com$ [NC]
RewriteRule ^(.*)$ http://inlinear.com/$1 [R=301,L] RewriteCond %{THE_REQUEST} ^./index.html
RewriteRule ^(.)index.html$ http://inlinear.com/ [R=301,L] 1. I redirect all URL-requests with www. to the non www-version...
2. all requests with "index.html" will be redirected to "domain.com/" My questions are: A) When linking from a page to my frontpage (home) the best practice is?: "http://domain.com/" the best and NOT: "http://domain.com/index.php" B) When linking to the index of a subfolder "http://domain.com/products/index.php" I should link also to: "http://domain.com/products/" and not put also the index.php..., right? C) When I define the canonical ULR, should I also define it just: "http://domain.com/products/" or in this case I should link to the definite file: "http://domain.com/products**/index.php**" Is A) B) the best practice? and C) ? Thanks for all replies! 🙂
Holger0 -
Transfer a Main Domain to a Sub-Domain
My IT department tells me they want to transfer my main site domain, which has been in existence since 1999 as an e-commerce site (maindomain.com) to a sub-domain (www2.maindomain.com) or a completely new domain (newdomain.net). This is because we are launching a new website and B2C e-commerce engine, but we still have to maintain the legacy B2B e-commerce engine which contains hard-coded URLs, and both systems can't use the same domain. I've been researching the issue across SEOmoz, but I haven't come across this exact type of scenario (mostly I've seen a sub-domain to new domain). I see major problems with their proposal, including negative SEO impact, loss of domain authority/ranking and issues with branding. Does anyone know the exact type of impact I can expect to see in this scenario and specific steps I should go about to minimize the impact? Btw, I will be using Danny Dover's guide on properly moving domains where appropriate. Thanks!
Technical SEO | | AscendLearning0 -
Is 301 redirecting all old URLS after a new site redesign to the root domain bad for SEO?
After a new site redesign ...would it hinder our rankings if we 301 redirected all old URLS that are returning 404 error codes to the root domain (home page) ? Would this be a good temporary solution until we are able to redirect the pages to the appropriate corresponding page? Thanks so much!
Technical SEO | | DCochrane0 -
Domain taken. Which is better? Using hypens or longer domain.
I am wanting to set up an e commerce site and the domain name that I want is taken. I am considering using a domain that has the main keyword I want to rank for as the domain. I have heard chatter of google penalizing these types of sites and it seems that it hasn't come about. This is something that I would like to test out. So if "electricscooters.com" is taken, should I use "electric-scooters.com" or "electricscooters4less.com" Just wondering if the hyphenated or the longer domain will rank higher. The site won't be spammy at all, I will carry a few different companies that offer similar products. So for this case, I would only sell scooters from a few different manufacturers. Feedback would be appreciated!
Technical SEO | | Dave_Whitty0 -
Redirect Multiple Domains
This is a follow-up question from one posted earlier this month. I can't linked to that because it's a private question so I'm trying to summarize it below. We have a number of domains – about 20 - (e.g. www.propertysharp.com) that point to our main domain ip adress (www.propertyshark.com) and share the same content. This is no black-hat strategy whatsoever, the domains were acquired several years ago in order to help people who mistyped the websites url to reach their desired destination. The question was whether to redirect them to our main domain or not. Pros were the reportedly millions of incoming links from these domains - cons was the fact that lots of issues regarding duplicate content could arise and we actually saw lots of some pages from these domains ranking in the search engines. We were recommended to redirect them, but to take it gradually. I have a simple question - what does gradually mean - one domain per week, per month?
Technical SEO | | propertyshark0 -
Switch domain from one account to another at domain registrar
We need to move our domain from one account to another at our domain registrar (which is Moniker). Both the "from" account and "to" account will be at Moniker. The "from" account currently has privacy settings enabled and we'd also put these in place for the "to" account. Has anyone done this and see any impact on SEO? Is there any big or common mistakes that I should be aware of? Thanks all!
Technical SEO | | evoNick0