Old domain still being crawled despite 301s to new domain
-
Hi there,
We switched from the domain X.com to Y.com in late 2013 and for the most part, the transition was successful. We were able to 301 most of our content over without too much trouble.
But when when I do a site:X.com in Google, I still see about 6240 URLs of X listed. But if you click on a link, you get 301d to Y. Maybe Google has not re-crawled those X pages to know of the 301 to Y, right? The home page of X.com is shown in the site:X.com results. But if I look at the cached version, the cached description will say :This is Google's cache of Y.com. It is a snapshot of the page as it appeared on July 31, 2014."
So, Google has freshly crawled the page. It does know of the 301 to Y and is showing that page's content. But the X.com home page still shows up on site:X.com. How is the domain for X showing rather than Y when even Google's cache is showing the page content and URL for Y?
There are some other similar examples. For instance, you would see a deep URL for X, but just looking at the <title>in the SERP, you can see it has crawled the Y equivalent. Clicking on the link gives you a 301 to the Y equivalent. The cached version of the deep URL to X also shows the content of Y.</p> <p>Any suggestions on how to fix this or if it's a problem. I'm concerned that some SEO equity is still being sequestered in the old domain.</p> <p>Thanks,</p> <p>Stephen</p></title>
-
Hey Stephen,
Did you ever solve this? I'm experiencing the exact same issues you've described above, so I'm curious if you ever figured it out.
Thanks.
-
If using a PHP CMS you will have a page # eg ?P123 this will make it so you can collect all of the old pages and have them redirect to the new ones perfectly.
Turn off all catching frist
A close friend of mine and I did this on his site which is quite large he got 30,000 URLs that were not pointing from his old domain to his new domain simply by adding the code below. If running WordPress and Nginx follow the steps below if you need to modify them so they correspond with your page format use
http://danielmiessler.com/blog/redirect-archives-pages/
Check out Yoast
https://yoast.com/wp-content/permalink-helper.php
&
https://yoast.com/change-wordpress-permalink-structure/
Sorry for being MIA,
Tom
-
Hi Matt, thanks for the reply. I can give this a test.
The puzzling thing here is that Google has re-crawled those old page URLs at the old domain. It is getting 301 re-directed to the new URL. The content that Google has cached for the old URL is from the new URL and has the new URL listed as the source of the cached page.
Stephen
-
Force a recrawl by pinging the pages.
Get as complete a list as possible of the pages that are indexed. Then submit them to PingFarm, 247pinger and RankonTop.
We had 2500+ pages indexed from an old development server and this removed almost all of them in 2-3 weeks.
-
Hi David, you can see my responses to these questions in my reply to Thomas. But the URLs from old X.com are being 301d correctly to y.com. For the vast majority of duplicate listings shown for site:x.com, you can click on them and they get 301d to y.com and the cached version of a page in site:x.com shows y.com content and even the y.com URL in the cached description.
-
I am going to assume the 301 redirects are working when you click on them?
Hi Thomas,
Yes, the 301s work and confirmed with http headers
_You need to go into Google Webmaster tools and tell Google that you have moved from domain A to domain B this will make Google bot crawl both sites vigorously looking for 301 redirects. _
The migration was before my time, but I think that this was done. If I try to use the change of address on the old site, X.com, I just get a "There is no change of address pending for your site." message with no control options (conversely, if I tried to do the same thing for the current Y.com, it would give me the full list of instructions on how to migrate my site.)
I want to be certain that you did not 301 redirect domain to domain and not page to page. So if site X has homepage, about page, whatever page, and site Y would contain the exact same pages or equivalent pages that you would have already 301 redirected page 2 page not just point the domain at the other domain is that right?
The 301s are at a page level at X.com to the same equivalent page at Y.com. The majority of pages have migrated over without obvious problems. But it's a little disturbing to see new pages of y.com which have never been part of x.com somehow make it into a site:x.com query and listed with an x.com domain to start the URL.
So just to recap, site:x.com shows long-tail pages that clearly belong to site:y.com and were never part of x.com. The <title>of some of the site:x.com pages. for instance, are definitely from y.com pages. For some reason, Google is associating these pages with the x.com domain.</p> <p>If you click on the cached version of a listing for site:x.com, the cached version will show the content and URL of Y.com/foo in the cached description header. Clicking on the actual link gets you 301d from X.com/foo to Y.com/foo. Both events indicate that the 301 is working and that Google is recognizing the 301.</p> <p>I don't know if this impacting our SERPs or not. If I do a very page-specific search for "blue widgets A, B, and C in Montana" for a page that is indexed in both site:x.com and site:y.com, you only see y.com's page which is expected behavior. You don't see x.com in the SERPs for that specific query. It's only if you do site:x.com "blue widgets A, B, and C in Montana" do you see the duplicate listing. But again, clicking on the URL that is shown results in a 301 to the proper y.com page.</p> <p>I can dig deeper with my developers, check logs, etc. But it's weird. It's almost like Google sees a URL for y.com and indexes that URL for x.com and y.com even though it knows that y.com is the dominant or real page (evidenced by cached data info). Everytime I click on an site:x.com link, I get 301 redirected properly to the y.com equivalent.</p></title>
-
Did you keep the page URL structure the same on the new site? If so, you can do a simple htaccess rule that will forwards all traffic from one domain, to the corresponding pages on the other domain. If not, then you will have to go through and check each one manually.
Have all the steps been covered? Did you resubmit a new sitemap? Specify in "fetch as Google" that all the new pages on the new site be crawled? If you are using a CMS, did you check to see if multiple versions of a certain page exist, and forget to redirect those?
Lastly, even though you still see references to the old domain, do the 301 redirects work? Once clicked do they send a user to the new domain?
-
I am going to assume the 301 redirects are working when you click on them?
Going on that assumption and please correct me if I am wrong. You need to go into Google Webmaster tools and tell Google that you have moved from domain A to domain B this will make Google bot crawl both sites vigorously looking for 301 redirects. Basically any changes that have occurred when the first crawl is completed after approximately 90 days tell Google to crawl it again. You have a fairly large site based on the amount of links you are discussing. You need to get Google to find everything. That is the best way to tell Google that you are changing domains.
https://support.google.com/webmasters/answer/55281
Two references I would use. Each contains a lot of good information.
https://support.google.com/webmasters/topic/6033102?hl=en&ref_topic=6029673
&
http://moz.com/community/q/how-to-keep-old-url-juice-during-site-switch
"But when I do a site:X.com in Google, I still see about 6240 URLs of X listed. But if you click on a link, you get 301d to Y. Maybe Google has not re-crawled those X pages to know of the 301 to Y, right? The home page of X.com is shown in the site:X.com results. But if I look at the cached version, the cached description will say :This is Google's cache of Y.com. It is a snapshot of the page as it appeared on July 31, 2014."
I want to be certain that you did not 301 redirect domain to domain and not page to page. So if site X has homepage, about page, whatever page, and site Y would contain the exact same pages or equivalent pages that you would have already 301 redirected page 2 page not just point the domain at the other domain is that right?
I hope this helps,
Thomas
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New domain wipes out domain authority
A client wanted to change their domain name, which we have now done. The site content itself is exactly the same. We put 301 redirect links in so that Google searchers would redirect from the old site to the new one. However Moz then said that it couldn't crawl the old domain because of the redirects and advised creating a brand new campaign for the new domain. We have done this but now Moz says that the domain authority of the new site is 2 (it was 14 on the old domain). Specifics are:
Technical SEO | | mfrgolfgti
old domain: https://ryemeadcleaning.co.uk
new domain: https://ryemeadgroup.co.uk So basically it seems like we're starting again from scratch with the new domain and all the SEO from the old domain has been lost? Have we done it wrong?0 -
Value of domain name for domain authority. Please help to figure out!
I am doing SEO for an appliance repair company. Their company website's domain doesn't have high authority, and I am going to increase that by link earning and content improving. I think a better domain name might also help me out. The current URL contain the word "appliance" but doesn't have "repair" in it. I am thinking a new domain that would contain both keywords will serve better. Could you please share with me your thought on this? Am I in the right direction, or not at all? I know Google penalizes mirror sites since this they are considered as duplicated content. I'll upload my content to the new domain and make the old one point to that new URL. I am wondering if canonical might help? Or 301 redirect will be a better solution? Any advise would be highly appreciated! Thank you!
Technical SEO | | kirupa0 -
Purchasing a domain to redirect to a new domain (note same industry) - Black hat or White hat technique?
Hi Everyone, Ok so here is my question. I have a client who sells gourmet tea and gourmet spices. She has a culinary blog. There is a culinary blog that just posted that the website will be shut down in the near future. It has 100% white hat links. Would it be considered black hat to buy the domain and redirect it to my clients blog which is also a culinary blog? I would really like to ask Matt Cutts this question. Does anyone know how to send him questions? Thanks Carla
Technical SEO | | Carla_Dawson0 -
Crawl completed but still says meta description missing
Before the last crawl I went through and made sure all meta descriptions were not missing. However, from the last crawl on the 26th July, it still has the same pages on the list and showed they were crawled as well? Any ideas on why they may still be showing as missing?
Technical SEO | | HamiltonIsland0 -
How Often is Site Crawled
Good morning- I saw some errors in my first crawl and immediately removed the pages from my website. I then re-created my XML sitemap and uploaded to Google. The question I have is will the site be crawled to recognize the changes in the next day or so? The pages were just placed on the site as test pages and never removed. The initial crawl that notified me it was done found the errors and were removed. Thanks for your help. Peter
Technical SEO | | VT_Pete0 -
Fixing Crawl Errors
Hi! I moved my Wordpress blog back in August, and lost much of my site traffic. I recently found over 1000 crawl errors in Webmaster Tools because some of my redirects weren't transferred, so we are working on fixing the errors and letting Google know. I'm wondering how long I should expect for Google to recognize that the errors have been fixed and for the traffic to start returning? Thanks! Jodi - momsfavoritestuff.com
Technical SEO | | JodiFTM0 -
Why is there duplicates of my domain
When viewing crawl diagnostics in SEOmoz I can see both "www.website.com" and a truncated version "website.com" is this normal and why is it showing (I do not have duplicates of my site on the server)? E.g.: http://www.klinehimalaya.com/
Technical SEO | | gorillakid
http://klinehimalaya.com/0 -
302 multiple domains...
Hello, I have a few domain names with orthographic variations that I'd like to redirect to my main site. The problem is my registrar (OVH) does only 302 redirects, so what are my options ? Can I keep a dozen 302's ? Do I have to change all their DNS (it's a load on my server...) ? Thanks for any ideas Johann.
Technical SEO | | JohannCR0