How Best to Handle Inherited 404s on Purchased Domain
-
We purchased a domain from another company and migrated our site over to it very successfully. However, we have one artifact of the original domain in that there was a page that was exploited by other sites on the web. This page allowed you to pass any URL to it and redirect to that URL (e.g. http://example.com/go/to/offsite_link.asp?GoURL=http://badactor.com/explicit_content).
This page does not exist on our site so the results always go to a 404 on our site. However, we find that crawlers are still attempting to access these invalid pages.
We have disavowed as many of the explicit sites as we can, but still some crawlers come looking for those links. We are considering blocking the redirect page in our robots.txt but we are concerned that the links will remain indexed but uncrawlable.
What's the best way to pull these pages from search engines and never have them crawled again?
UPDATE: Clarifying that what we're trying to do it get search engines to just never try to get to these pages. We feel the fact they're even wasting their time on getting a 404 is what we're trying to avoid. Is there any reason we shouldn't just block these in our robots.txt?
-
@gastonriera calm down mate. We have actually tested this at not seen any negative effect on any site we have done it on. It is the "easiest" option, but it won't cause the death and destruction your comment implies. Good day sir.
-
Hi there,
I'm considering that you have over 500k URLs, to be worrying about crawl efficiency. If you have less than that, please don't worry.
Having 404s is completely fine, and google will eventually lower their crawl frequency to those pages.
Blocking them in robots.txt will cause to google stop crawling them, but never to never remove them from the index.
My advice here: don't block them in robots.txtAs Rajesh pointed out, you could force those 404s into 410 to tell Google that they are gone forever. Yet, Google said that they treat 404s and 410s as the same.
John Mueller said over a year ago that 4xx status codes don't incur in crawl wastage. You can check it our in these Webmasters hangout notes - DeepcrawlHope it helps,
Best luck.
Gaston -
FOR THE LOVE OF GOD DONT REDIRECT 404s TO THE HOME!
This is terrible advice. Doing that you'll turn those 404s into soft 404s, making them more problematic than ever.
-
I would actually recommend redirecting it to the homepage. If you have a Wordpress website and a bunch of 404 pages, you can install a free plugin called "All 404 to Homepage" and this will solve the problem. I would, however, recommend that if you have replacement pages or pages covering similar content, that you redirect those to the corresponding replacement page.
-
You need to do one thing with those 404 pages. Move them as 410 status code. Redirection is not good practice for the same.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Domain dominance
I've just started to work for a company who've purchased masses of domains with every conceivable permutation based on all their products with every extension possible e.g .biz . eu. .net (including .co.uk and .com of course). I have two questions: 1. Is it worth keeping all these (they want to add more) domains or let them expire? 2. All the purchased domains are online - is there any point (they redirect with a 301)?
Intermediate & Advanced SEO | | LJHopkins0 -
Is there a way to increase domain authority?
Dear all, when I see moz analytics for my blog irctcloginindia.co.in, it is legging behind only in terms of diomain authority when compared to my competitors. Because of which it is ranking low. Is there any short cut or fast method using which I can increase the authority for my domain.
Intermediate & Advanced SEO | | irctclogin0 -
What is the best way to track rank?
I'm curious what tools and tracking mechanisms everyone uses to track rank, especially with all of the changes in the last year. I've read a ton of articles that say even in 2014, rank will continue to matter, but besides using the keyword rank tool in Moz, I was hoping to learn what everyone else uses to track rank on a keyword and page level (are those two separate things?).
Intermediate & Advanced SEO | | RachelEm0 -
Domain Name Redirect Question
My agency just built a new website for a client who is a franchisee. It's not launched yet - it's currently under an IP address. I suggested to client that he buy a keyword-rich domain name for it, which he did. Then he found out that the franchisor will not allow it to be his main domain name. They want him to use a domain name with the franchisor name in it. But they WILL allow him to put a 301 redirect on that franchisor-approved domain name, and redirect it to his keyword-rich domain name. He is interested in having my agency perform an SEO Campaign for this new website. But would SEO and link marketing work for a website that has a new non-keyword domain name that 301 redirects to a new keyword-rich domain name?
Intermediate & Advanced SEO | | netsites0 -
Subdomain, subfolder or separate domains?
Hi Mozzers, We're in the process of re-developing and redesigning several of our websites, and moving them all onto the same content management system. At the moment, although the websites are all under the same brand and roughly the same designs, because of various reasons they all either live on a separate domain to the main website, or are on a subdomain. Here's a list of what we have (and what we're consolidating): Main site - http://www.frenchentree.com/ Property database - http://france-property.frenchentree.com/ (subdomain) Forum - http://www.france-forum-frenchentree.com/ (separate domain) Classified ads - http://www.france-classified-ads-frenchentree.com/ (separate domain) My question to you lovely people is: should we take this opportunity through the redevelopment of the CMS to put everything into subfolders of the main domain? Keep things as they are? Put each section onto a subdomain? What's best from an SEO perspective? For information - the property database was put onto a subdomain as this is what we were advised to do by the developers of the system. We're starting to question this decision though, as we very rarely see subdomains appear in SERPs for any remotely competitive search terms. Our SEO for the property database is fairly non-existent, and only ever really appears in SERPs for brand related keywords. For further info - the forum and classifieds were under a separate brand name previously, so keeping them on separate domains felt correct at that time. However, with the redevelopment of our sites, it seems to make more sense to either put them on subdomains or subfolders of the main site. Our SEO for the forum is pretty strong, though has dwindled in the last year or so. Any help/advice would be very much appreciated. Thanks Matt
Intermediate & Advanced SEO | | Horizon0 -
Should we move a strong category page, or the whole domain to new domain?
We are debating moving a strong category page (and subcategory, product pages) from our current older domain to a new domain vs just moving the whole domain. The older domain has DA 40+, and the category page has PA 40+. Anyone with experience on how much PR etc will get passed to a virgin domain if we just redirect olddomain/strongcategorypage/ to newdomain.com? If the answer is little to none, we might consider just moving the whole site since the other categories are not that strong anyway. We will use 301 approach either way. Thanks!
Intermediate & Advanced SEO | | Durand0 -
Age of a re-directed domain same as age of a static domain?
I know domain age plays a role in SEO--but, I am wondering if a domain is set up to 301 re-direct to another domain if it builds the same amount of authority over time as the static domain--just looking at age as a ranking factor, not links accumulated over time.
Intermediate & Advanced SEO | | nicole.healthline0 -
Exact match domain or root domain for speedy SEO?
I am doing SEO for a website that has constantly rotating and only temporarily pertinent subjects on it. Let's say these information and subject cycles go for about 6 months. Assuming this would it be more effective to optimize exact match domains for each 6 month cycle or make a main domain with a few of the keywords and just target a page for each roaming subject? Advantage of the subject is I get domain authority to feed off of, advantage of the exact match is, of course exact match domains are a powerful tool to rank highly and it is only a medium competitive market, usually about 40 domain and page authority. What do you guys think? Do you have any techniques to dominate temporary and rotating markets?
Intermediate & Advanced SEO | | MarloSchneider0