How Best to Handle Inherited 404s on Purchased Domain
-
We purchased a domain from another company and migrated our site over to it very successfully. However, we have one artifact of the original domain in that there was a page that was exploited by other sites on the web. This page allowed you to pass any URL to it and redirect to that URL (e.g. http://example.com/go/to/offsite_link.asp?GoURL=http://badactor.com/explicit_content).
This page does not exist on our site so the results always go to a 404 on our site. However, we find that crawlers are still attempting to access these invalid pages.
We have disavowed as many of the explicit sites as we can, but still some crawlers come looking for those links. We are considering blocking the redirect page in our robots.txt but we are concerned that the links will remain indexed but uncrawlable.
What's the best way to pull these pages from search engines and never have them crawled again?
UPDATE: Clarifying that what we're trying to do it get search engines to just never try to get to these pages. We feel the fact they're even wasting their time on getting a 404 is what we're trying to avoid. Is there any reason we shouldn't just block these in our robots.txt?
-
@gastonriera calm down mate. We have actually tested this at not seen any negative effect on any site we have done it on. It is the "easiest" option, but it won't cause the death and destruction your comment implies. Good day sir.
-
Hi there,
I'm considering that you have over 500k URLs, to be worrying about crawl efficiency. If you have less than that, please don't worry.
Having 404s is completely fine, and google will eventually lower their crawl frequency to those pages.
Blocking them in robots.txt will cause to google stop crawling them, but never to never remove them from the index.
My advice here: don't block them in robots.txtAs Rajesh pointed out, you could force those 404s into 410 to tell Google that they are gone forever. Yet, Google said that they treat 404s and 410s as the same.
John Mueller said over a year ago that 4xx status codes don't incur in crawl wastage. You can check it our in these Webmasters hangout notes - DeepcrawlHope it helps,
Best luck.
Gaston -
FOR THE LOVE OF GOD DONT REDIRECT 404s TO THE HOME!
This is terrible advice. Doing that you'll turn those 404s into soft 404s, making them more problematic than ever.
-
I would actually recommend redirecting it to the homepage. If you have a Wordpress website and a bunch of 404 pages, you can install a free plugin called "All 404 to Homepage" and this will solve the problem. I would, however, recommend that if you have replacement pages or pages covering similar content, that you redirect those to the corresponding replacement page.
-
You need to do one thing with those 404 pages. Move them as 410 status code. Redirection is not good practice for the same.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why some domains and sub-domains have same DA, but some others don't?
Hi I noticed for some blog providers in my country, which provide a sub-domian address for their blogs. the sub-domain authority is exactly as the main domain. Whereas, for some other blog providers every subdomain has its different and lower authority. for example "ffff.blog.ir" and "blog.ir" both have domain authority of 60. It noteworthy to mention that the "ffff.blog.ir" does not even exist! This is while mihanblog.com and hfilm.mihanblog.com has diffrent page authority.
Intermediate & Advanced SEO | | rayatarh5451230 -
Domain Migration Hell!
5 weeks ago we migrated our site to a new domain. We also installed an SSL certificate on the new domain. The new domain was purchased 5 years ago but we only used it as a redirect address. It was more consistent with our brand so we decided to migrate to it. Great care was taken setting up page to page redirects. A formal domain change request was made to Google. In fact the move was implemented with only a handful of broken links on a 500 page site. Those links were quickly fixed. Our traffic declined from about 350 visitors a week to as low as 40 visitors the first full week after the move. Now the number of organic Google visits is up to 80, a drop of 75% !!! All except 20 (out of 500) pages are reindexed on Google Search Console. MOZ domain authority for the new domain has climbed from 5 to about 12. The old domain had a DA of 23. In Google Search Console hundreds of "URL Not Allowed" errors are the site map for our previous domain that redirects to our new domain. Attached please see image of this. The site map for the new domain appears normal, but about 160 pages are indexed that are not in the sitemap. I wonder if these two issues have somehow contributed to the drop in ranking. I have included images showing GCT for the 2 domains. I posted on MOZ a month ago and was told it just might take time. No improvement and now I am wonder if there is not some issue with the sitemaps causing havoc. Are traffic is down more than 80%. This does not seem normal. Any advice? Any suggestions as to how to expedite recovery? Thanks,
Intermediate & Advanced SEO | | Kingalan1
Alan0 -
Purchased Domain Not Related to Vertical and Moved Questions
Hello, We used to reside on a longer domain name character wise but had an opportunity to purchase a three letter acronym before my time. We advertise heavily on TV and other traditional media channels so it was simpler for individuals to remember. After purchasing this domain we moved over from the old one to the new one. We followed all of your standard protocols (301s, change of address in Webmaster Tools, new sitemap, etc, etc.) Google and Bing both index the new site and there isn't a significant issue there but we're having incredible difficulty in ranking for any of our core terms. We're the largest company in our space but continue to rank for terms that have nothing to do with our vertical. This is due to the fact that the site used to be owned by a company that is completely separate from ours. The site that we have today contains none of the old content but it does have links pointing to it from similar sites from that vertical. Bing was incredibly helpful and had indicated it's these links that's potentially causing us the issue in that search engines are seeing two different verticals over time on a domain. It's been a year since this took place and it seems that the only recommendation is to contact the non related sites to remove links or disavow. Bing had indicated that disavowing was not as relevant as getting the links removed. Any other thoughts?
Intermediate & Advanced SEO | | seo--team-jlck0 -
Moving low ranking domain
I have a website, that I rewrote great content for, but I recently found that there are many, many links going to the subdomain that may be pulling it down. Has anyone had experience taking down a site and then moving the content to a new site? Will it be considered duplicate content if you completely take the old site down and use rel="canonical" on new site pages? I don't want to lose the good content, but I cannot have it on the current URL with all the bad backlinking (it's a complicated situation, as I need to keep those backlinks which are affiliates). Thank you.
Intermediate & Advanced SEO | | RoxBrock0 -
Redirect old .net domain to new .com domain
I have a quick question that I think I know the answer to but I wanted to get some feedback to make sure or see if there's additional feedback. The long and short of it is that I'm working with a site that currently has a .net domain that they've been running for 6 years. They've recently bought a .com of the same name as well. So the question is: I think it's obviously preferable to keep the .net and just direct the .com to it. However, if they would prefer to have the .com domain, is 301'ing the .net to the .com going to lose a lot of the equity they've built up in the site over the past years? And are there any steps that would make such a move easier? Also, if you have any tips or insight just into a general transition of this nature it would be much appreciated. Thanks!
Intermediate & Advanced SEO | | BrandLabs0 -
Should I 301 a penalized domain to another domains subfolder?
I have a niche domain seems to have been hit by Penguin. It had very good rankings before the update, and I think at least a good part of the penalty might be due to overoptimized anchor text. So here is the question; If I decide to take this site down, should I 301 the entire domain to a relevant sub-folder of another site? i.e comtemporaryfurniture.com to domain.com/category/modern-furniture.html Will the penalty get passed onto the new domain? If the penalty is partly due to anchor text, then pointing it to another site's subfolder would mean the tartget URL has more varied anchor text and could boost rankings.
Intermediate & Advanced SEO | | inhouseseo0 -
Improve Domain Trust
Hey there, I need to improve the Domain trust of my website. I've seen the blogitems of Rand but I still have questions. If I want to increase the Domain Trust..I "simpely" have to get links from websites with a high Domain Trust. This will eventually improve my Domain trust. But what are the effects if I do this by reciprocal linking. Does this eliminates the trust that comes from the other website.
Intermediate & Advanced SEO | | PlusPort1 -
Redirect on exact match domain to Brand domain question :)
Hi, If I have a website with the domain crazysocks.co.uk and a title tag 'black socks' would I see any benefit redirecting blacksocks.co.uk to crazysocks.co.uk, to give my keyword 'black socks' a boost in the SE's from the EMD. I see it loads where an EMD is indexed for its term but when you click the result it redirects to a branded domain. I personally cant see this being true but wanted to double check.
Intermediate & Advanced SEO | | activitysuper0