How Best to Handle Inherited 404s on Purchased Domain
-
We purchased a domain from another company and migrated our site over to it very successfully. However, we have one artifact of the original domain in that there was a page that was exploited by other sites on the web. This page allowed you to pass any URL to it and redirect to that URL (e.g. http://example.com/go/to/offsite_link.asp?GoURL=http://badactor.com/explicit_content).
This page does not exist on our site so the results always go to a 404 on our site. However, we find that crawlers are still attempting to access these invalid pages.
We have disavowed as many of the explicit sites as we can, but still some crawlers come looking for those links. We are considering blocking the redirect page in our robots.txt but we are concerned that the links will remain indexed but uncrawlable.
What's the best way to pull these pages from search engines and never have them crawled again?
UPDATE: Clarifying that what we're trying to do it get search engines to just never try to get to these pages. We feel the fact they're even wasting their time on getting a 404 is what we're trying to avoid. Is there any reason we shouldn't just block these in our robots.txt?
-
@gastonriera calm down mate. We have actually tested this at not seen any negative effect on any site we have done it on. It is the "easiest" option, but it won't cause the death and destruction your comment implies. Good day sir.
-
Hi there,
I'm considering that you have over 500k URLs, to be worrying about crawl efficiency. If you have less than that, please don't worry.
Having 404s is completely fine, and google will eventually lower their crawl frequency to those pages.
Blocking them in robots.txt will cause to google stop crawling them, but never to never remove them from the index.
My advice here: don't block them in robots.txtAs Rajesh pointed out, you could force those 404s into 410 to tell Google that they are gone forever. Yet, Google said that they treat 404s and 410s as the same.
John Mueller said over a year ago that 4xx status codes don't incur in crawl wastage. You can check it our in these Webmasters hangout notes - DeepcrawlHope it helps,
Best luck.
Gaston -
FOR THE LOVE OF GOD DONT REDIRECT 404s TO THE HOME!
This is terrible advice. Doing that you'll turn those 404s into soft 404s, making them more problematic than ever.
-
I would actually recommend redirecting it to the homepage. If you have a Wordpress website and a bunch of 404 pages, you can install a free plugin called "All 404 to Homepage" and this will solve the problem. I would, however, recommend that if you have replacement pages or pages covering similar content, that you redirect those to the corresponding replacement page.
-
You need to do one thing with those 404 pages. Move them as 410 status code. Redirection is not good practice for the same.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Brightcove SEO Best Practices?
My company has decided to go with Brightcove as a video platform so we can better monetize all of the video content we create and better customize the experience as well. We have a pretty decent YouTube presence, and I won't let them stop using that because it would totally alienate us from part of our audience. So I was hoping someone could help me with the following: Are we able to keep videos hosted on YouTube as well as Brightcove without any risk of duplicate content? If we use the Brightcove player to embed videos in our on-site content, are we hindering potential organic search visibility? On the embeds, it's looking like it's using an iframe in our content (https://www.leafly.com/news/cannabis-101/sativa-indica-and-hybrid-whats-the-difference-between-cannabis-ty) - We're using a Brightcove WP plugin for the embed, but I was wondering if anyone had suggestions on a better way to implement/if this is even an issue at all. Are there any other general best practices/insights anyone has working with this platform? I found this article on their site, but I was wondering if there was anything else I should consider. Thank you in advance for any insights/answers!
Intermediate & Advanced SEO | | davidkaralisjr0 -
Domain Change Before or After Site Revamp?
In the last year traffic to our site has dropped in half and ranking has dropped significantly. Very little no content has been added in that time. We would now like to improve ranking by adding new content. 2 domains effectively exist for the site. The existing domain is www.nyc-officespace-leader.com. But www.metro-manhattan.com redirects to www.nyc-officespace-leader.com. Our company is Metro Manhattan Office Space, Inc.. We registered www.metro-manhattan.com and created the redirect to www.nyc-officespace-leader.com in 2012. www.nyc-officespace-leader.com was registered in 2006. Many links to the site show www.metro-manhattan.com and I believe this may be a source of confusion for Google. Would it be best to make the domain consistent at this time by redirecting it once and for all and to do so before adding new content? If this is done correctly can we avoid taking a hit on ranking? Note: -www.nyc-officespace-leader.com is the old domain.
Intermediate & Advanced SEO | | Kingalan1
-www.metro-manhattan is the new domain but has existed since 2012 and has been redirecting to the old domain since then
-The company name is Metro Manhattan Office Space (similar in branding to the new domain) Am I correct in assuming that having the 2 domains may be causing issues with Google involving domain authority? Change the domain before adding content or add content before?0 -
Should I Forward Niche Site To Main Domain?
I have a niche site that was hit by penguin. I was never hit with a manual penalty, but it is obvious the algo update hit the site. I have submitted a disavow file, but it doesn't seem to have made an impact. The site does have good rankings on Yahoo and Bing. Should I close this site and 301 to the category page on my main site (i.e tshirts.com forwarded to mainsite.com/tshirts.html)? The main site had a drop after penguin, but is now receiving more traffic than pre-Penguin. The impact is mainly keyword specific (i.e keywords that appeared in anchor text dropped). Appreciate thoughts.
Intermediate & Advanced SEO | | inhouseseo0 -
Redirect ruined domain to new domain without passing link juice
A new client has a domain which has been hammered by bad links, updates etc and it's basically on its arse because of previous SEO guys. They have various domains for their business (brand.com, brand.co.uk) and want to use a fresh domain and take it from there. Their current domain is brand.com (the ruined one). They're not bothered about the rankings for brand.com but they want to redirect brand.com to brand.co.uk so that previous clients can find them easily. Would a 302 redirect work for this? I don't want to set up a 301 redirect as I don't want any of the crappy links pointing across. Thanks!
Intermediate & Advanced SEO | | jasonwdexter0 -
Domain Forwarding
I have a client who previously registered 20 unique domain names that tied to their company name and services. They use all of these domains to forward to their main website to try and capture additional traffic. Would you suggest that we remove all of the domains by 301 redirecting them all the the main website? I am trying to find a good article that shows the implications on SEO by using many domains that forward to a main website. Any suggestions would be greatly appreciated.
Intermediate & Advanced SEO | | Prager0 -
Should I 301 a penalized domain to another domains subfolder?
I have a niche domain seems to have been hit by Penguin. It had very good rankings before the update, and I think at least a good part of the penalty might be due to overoptimized anchor text. So here is the question; If I decide to take this site down, should I 301 the entire domain to a relevant sub-folder of another site? i.e comtemporaryfurniture.com to domain.com/category/modern-furniture.html Will the penalty get passed onto the new domain? If the penalty is partly due to anchor text, then pointing it to another site's subfolder would mean the tartget URL has more varied anchor text and could boost rankings.
Intermediate & Advanced SEO | | inhouseseo0 -
Exact match domain names
Hello, Someone approached a client of mine to sell a exact match domain name for a very competitive and high converting keyword. Would this be of any use and what are the best tactics to employ if it is purchased? I was of the opinion that the 'power' of exact match domain names are dying fast but would be interested to hear what people with experience in this think and what they have done with them (i.e. set-up a website on that domain or re-directed it)? Thanks, Rikki
Intermediate & Advanced SEO | | RikkiD220 -
301 from penalized domain to new domain
I have a client whose site isn't necessarily penalized since they still show for many terms in the SERPS, however at one point they did an xrummer blast of 13,000 links for two anchor texts they were trying to rank for. They have purchased a new domain and have gone white hat and want to 301 some of the old site to the new purely for the users sake so past visitors still find them at t the new location. Will creating 301 redirects pass on to the new domain any bad Karma from the old one in Google's eyes? Thanks for the help.
Intermediate & Advanced SEO | | JoshGill270