Old domain still being crawled despite 301s to new domain
-
Hi there,
We switched from the domain X.com to Y.com in late 2013 and for the most part, the transition was successful. We were able to 301 most of our content over without too much trouble.
But when when I do a site:X.com in Google, I still see about 6240 URLs of X listed. But if you click on a link, you get 301d to Y. Maybe Google has not re-crawled those X pages to know of the 301 to Y, right? The home page of X.com is shown in the site:X.com results. But if I look at the cached version, the cached description will say :This is Google's cache of Y.com. It is a snapshot of the page as it appeared on July 31, 2014."
So, Google has freshly crawled the page. It does know of the 301 to Y and is showing that page's content. But the X.com home page still shows up on site:X.com. How is the domain for X showing rather than Y when even Google's cache is showing the page content and URL for Y?
There are some other similar examples. For instance, you would see a deep URL for X, but just looking at the <title>in the SERP, you can see it has crawled the Y equivalent. Clicking on the link gives you a 301 to the Y equivalent. The cached version of the deep URL to X also shows the content of Y.</p> <p>Any suggestions on how to fix this or if it's a problem. I'm concerned that some SEO equity is still being sequestered in the old domain.</p> <p>Thanks,</p> <p>Stephen</p></title>
-
Hey Stephen,
Did you ever solve this? I'm experiencing the exact same issues you've described above, so I'm curious if you ever figured it out.
Thanks.
-
If using a PHP CMS you will have a page # eg ?P123 this will make it so you can collect all of the old pages and have them redirect to the new ones perfectly.
Turn off all catching frist
A close friend of mine and I did this on his site which is quite large he got 30,000 URLs that were not pointing from his old domain to his new domain simply by adding the code below. If running WordPress and Nginx follow the steps below if you need to modify them so they correspond with your page format use
http://danielmiessler.com/blog/redirect-archives-pages/
Check out Yoast
https://yoast.com/wp-content/permalink-helper.php
&
https://yoast.com/change-wordpress-permalink-structure/
Sorry for being MIA,
Tom
-
Hi Matt, thanks for the reply. I can give this a test.
The puzzling thing here is that Google has re-crawled those old page URLs at the old domain. It is getting 301 re-directed to the new URL. The content that Google has cached for the old URL is from the new URL and has the new URL listed as the source of the cached page.
Stephen
-
Force a recrawl by pinging the pages.
Get as complete a list as possible of the pages that are indexed. Then submit them to PingFarm, 247pinger and RankonTop.
We had 2500+ pages indexed from an old development server and this removed almost all of them in 2-3 weeks.
-
Hi David, you can see my responses to these questions in my reply to Thomas. But the URLs from old X.com are being 301d correctly to y.com. For the vast majority of duplicate listings shown for site:x.com, you can click on them and they get 301d to y.com and the cached version of a page in site:x.com shows y.com content and even the y.com URL in the cached description.
-
I am going to assume the 301 redirects are working when you click on them?
Hi Thomas,
Yes, the 301s work and confirmed with http headers
_You need to go into Google Webmaster tools and tell Google that you have moved from domain A to domain B this will make Google bot crawl both sites vigorously looking for 301 redirects. _
The migration was before my time, but I think that this was done. If I try to use the change of address on the old site, X.com, I just get a "There is no change of address pending for your site." message with no control options (conversely, if I tried to do the same thing for the current Y.com, it would give me the full list of instructions on how to migrate my site.)
I want to be certain that you did not 301 redirect domain to domain and not page to page. So if site X has homepage, about page, whatever page, and site Y would contain the exact same pages or equivalent pages that you would have already 301 redirected page 2 page not just point the domain at the other domain is that right?
The 301s are at a page level at X.com to the same equivalent page at Y.com. The majority of pages have migrated over without obvious problems. But it's a little disturbing to see new pages of y.com which have never been part of x.com somehow make it into a site:x.com query and listed with an x.com domain to start the URL.
So just to recap, site:x.com shows long-tail pages that clearly belong to site:y.com and were never part of x.com. The <title>of some of the site:x.com pages. for instance, are definitely from y.com pages. For some reason, Google is associating these pages with the x.com domain.</p> <p>If you click on the cached version of a listing for site:x.com, the cached version will show the content and URL of Y.com/foo in the cached description header. Clicking on the actual link gets you 301d from X.com/foo to Y.com/foo. Both events indicate that the 301 is working and that Google is recognizing the 301.</p> <p>I don't know if this impacting our SERPs or not. If I do a very page-specific search for "blue widgets A, B, and C in Montana" for a page that is indexed in both site:x.com and site:y.com, you only see y.com's page which is expected behavior. You don't see x.com in the SERPs for that specific query. It's only if you do site:x.com "blue widgets A, B, and C in Montana" do you see the duplicate listing. But again, clicking on the URL that is shown results in a 301 to the proper y.com page.</p> <p>I can dig deeper with my developers, check logs, etc. But it's weird. It's almost like Google sees a URL for y.com and indexes that URL for x.com and y.com even though it knows that y.com is the dominant or real page (evidenced by cached data info). Everytime I click on an site:x.com link, I get 301 redirected properly to the y.com equivalent.</p></title>
-
Did you keep the page URL structure the same on the new site? If so, you can do a simple htaccess rule that will forwards all traffic from one domain, to the corresponding pages on the other domain. If not, then you will have to go through and check each one manually.
Have all the steps been covered? Did you resubmit a new sitemap? Specify in "fetch as Google" that all the new pages on the new site be crawled? If you are using a CMS, did you check to see if multiple versions of a certain page exist, and forget to redirect those?
Lastly, even though you still see references to the old domain, do the 301 redirects work? Once clicked do they send a user to the new domain?
-
I am going to assume the 301 redirects are working when you click on them?
Going on that assumption and please correct me if I am wrong. You need to go into Google Webmaster tools and tell Google that you have moved from domain A to domain B this will make Google bot crawl both sites vigorously looking for 301 redirects. Basically any changes that have occurred when the first crawl is completed after approximately 90 days tell Google to crawl it again. You have a fairly large site based on the amount of links you are discussing. You need to get Google to find everything. That is the best way to tell Google that you are changing domains.
https://support.google.com/webmasters/answer/55281
Two references I would use. Each contains a lot of good information.
https://support.google.com/webmasters/topic/6033102?hl=en&ref_topic=6029673
&
http://moz.com/community/q/how-to-keep-old-url-juice-during-site-switch
"But when I do a site:X.com in Google, I still see about 6240 URLs of X listed. But if you click on a link, you get 301d to Y. Maybe Google has not re-crawled those X pages to know of the 301 to Y, right? The home page of X.com is shown in the site:X.com results. But if I look at the cached version, the cached description will say :This is Google's cache of Y.com. It is a snapshot of the page as it appeared on July 31, 2014."
I want to be certain that you did not 301 redirect domain to domain and not page to page. So if site X has homepage, about page, whatever page, and site Y would contain the exact same pages or equivalent pages that you would have already 301 redirected page 2 page not just point the domain at the other domain is that right?
I hope this helps,
Thomas
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to get google to forget my old but still working page and list my new fully optimized page for a keyword?
Hi There! (i am beginner in seo) I have dynamic and static pages on our site. I created a static page for a specific keyword. Fully optimized it, (h1, alt, metas, etc.....maybe too optimized). My problem is that this page is alive for weeks, checked it in GWT and it is in robots.txt, google sees it, and indexed it. BUT whenewer i do a search for that keyword, we still appear with the dynamically created link in the google listings. How could i "redirect" google, if sy make a search for that keyword than shows our optimized page? Is there a tool for that? I cant delete the dynamic page... Any ideas? Thx Andrew
Technical SEO | | Neckermann0 -
Clients domain expired - rankings lost - repurchased domain - what next?
Its only been 10 days and i have repurchased the domain name/ renewed. The who is info, website and contact information is all still the same. However we have lost all rankings and i am hoping that our top rankings come back. Does anyone have experience with such a crappy situation?
Technical SEO | | waqid0 -
Should we re-use our old re-directed domain for a new wesbite ?
Hi One our clients has an old domain that has been redirected to another website of his. Now he is asking us to build a new website for that domain and direct it back. This is new website will be very relative to it's own old content and where it has been re-directed recently. I guess the only benefit of this would be taking advantage of the age of this domain. Do you recommend doing that or getting him a new domain ?
Technical SEO | | Dynamo-Web0 -
An EMD with top level domain of another country. Still useful ?
Normally, EMDs (with couple of keywords in it) has more advantage over other domain names ( I understand other factors matters too & I am aware of EMD update). But what if there is an EMD with couple of keywords but top level domain is of another country. Confused? see ex - shoesinsydney.co.uk Will it STILL have a natural advantage over other domains (with no keywords in them)?
Technical SEO | | Personnel_Concept0 -
301'ing domain to an addon domain
My googlefu failed me in finding this... How to 301 a domain to an addon domain? Domain structure is as follows: http://addondomain.maindomain.com/ http://www.maindomain.com/addondomain/ http://www.addondomain.com/ <--(addon domain has its own domain as well) I want main domain to all point to the addon domain like so: http://www.maindomain.com/ --> http://www.addondomain.com/
Technical SEO | | JasonJackson0 -
Crawl Errors
Okay, I was just in my Google Webmaster Tools and was looking at some of the stats. I have 1354 "not found" pages google says. Many of these URL's are bizarre. I don't know what they are. Others I do know. What should I do about this? Especially all the URL's I don't even know what they are?
Technical SEO | | azguy0 -
New website with slightly new urls
Hi we recently designed our website in work and changed some of the urls. the old site used to be http://www.example.ie/contact-us.htm now it's is http://example.ie/get-in-touch The problem we are having is with sitelinks (the ones auto generate in the serp) ie: about, contact us, team etc etc. Once cliked on, these OLD links are all going to 404 pages because of the change of url. Help with this would be greatly appreciated - I was thinking of blocking these old sitelinks in google web master.
Technical SEO | | GlenBOB0 -
Using hyphenated sub-domains or non-hyphenated sub-domains? What is the question! I Any takers?
For our corporate business level domain, we are exploring using a hyphenated sub-domain foir a project. Something like www.go-figure.extreme.com I thought from a user perspective it seems cluttered. The domain length might also be an issue with the new Algorithm big G has launched in recent past. I know with past experience, hyphenated domains usually take longer to index, as they are used by spammers more frequently and can take longer to get out of the supplementary index. Our company site has over 90 million viewers / year, so our brand is well established and traffic isn't an issue. This is for a corporate level project and I didn't have the answer! Will this work? anyone have any experience testing this. Any thoughts will help! Thanks, Rob
Technical SEO | | RobMay0