How Best to Handle Inherited 404s on Purchased Domain
-
We purchased a domain from another company and migrated our site over to it very successfully. However, we have one artifact of the original domain in that there was a page that was exploited by other sites on the web. This page allowed you to pass any URL to it and redirect to that URL (e.g. http://example.com/go/to/offsite_link.asp?GoURL=http://badactor.com/explicit_content).
This page does not exist on our site so the results always go to a 404 on our site. However, we find that crawlers are still attempting to access these invalid pages.
We have disavowed as many of the explicit sites as we can, but still some crawlers come looking for those links. We are considering blocking the redirect page in our robots.txt but we are concerned that the links will remain indexed but uncrawlable.
What's the best way to pull these pages from search engines and never have them crawled again?
UPDATE: Clarifying that what we're trying to do it get search engines to just never try to get to these pages. We feel the fact they're even wasting their time on getting a 404 is what we're trying to avoid. Is there any reason we shouldn't just block these in our robots.txt?
-
@gastonriera calm down mate. We have actually tested this at not seen any negative effect on any site we have done it on. It is the "easiest" option, but it won't cause the death and destruction your comment implies. Good day sir.
-
Hi there,
I'm considering that you have over 500k URLs, to be worrying about crawl efficiency. If you have less than that, please don't worry.
Having 404s is completely fine, and google will eventually lower their crawl frequency to those pages.
Blocking them in robots.txt will cause to google stop crawling them, but never to never remove them from the index.
My advice here: don't block them in robots.txtAs Rajesh pointed out, you could force those 404s into 410 to tell Google that they are gone forever. Yet, Google said that they treat 404s and 410s as the same.
John Mueller said over a year ago that 4xx status codes don't incur in crawl wastage. You can check it our in these Webmasters hangout notes - DeepcrawlHope it helps,
Best luck.
Gaston -
FOR THE LOVE OF GOD DONT REDIRECT 404s TO THE HOME!
This is terrible advice. Doing that you'll turn those 404s into soft 404s, making them more problematic than ever.
-
I would actually recommend redirecting it to the homepage. If you have a Wordpress website and a bunch of 404 pages, you can install a free plugin called "All 404 to Homepage" and this will solve the problem. I would, however, recommend that if you have replacement pages or pages covering similar content, that you redirect those to the corresponding replacement page.
-
You need to do one thing with those 404 pages. Move them as 410 status code. Redirection is not good practice for the same.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301 redirecting staff Domain to Company Domain
My colleague owns a domain (A) for about 10 years that he does not use. The domain's content is the same as my company's website (B) content.
Intermediate & Advanced SEO | | khi5
Question: Can I 301 redirect domain A to domain B's homepage or is it better he just closes down his website since this would not be SEO best practices? thank you0 -
Purchased Domain Not Related to Vertical and Moved Questions
Hello, We used to reside on a longer domain name character wise but had an opportunity to purchase a three letter acronym before my time. We advertise heavily on TV and other traditional media channels so it was simpler for individuals to remember. After purchasing this domain we moved over from the old one to the new one. We followed all of your standard protocols (301s, change of address in Webmaster Tools, new sitemap, etc, etc.) Google and Bing both index the new site and there isn't a significant issue there but we're having incredible difficulty in ranking for any of our core terms. We're the largest company in our space but continue to rank for terms that have nothing to do with our vertical. This is due to the fact that the site used to be owned by a company that is completely separate from ours. The site that we have today contains none of the old content but it does have links pointing to it from similar sites from that vertical. Bing was incredibly helpful and had indicated it's these links that's potentially causing us the issue in that search engines are seeing two different verticals over time on a domain. It's been a year since this took place and it seems that the only recommendation is to contact the non related sites to remove links or disavow. Bing had indicated that disavowing was not as relevant as getting the links removed. Any other thoughts?
Intermediate & Advanced SEO | | seo--team-jlck0 -
Unique domains vs. single domain for UGC sites?
Working on a client project - a UGC community that has a DTC model as well as a white label model. Is it categorically better to have them all under the same domain? Trying to figure which is better: XXX,XXX pages on one site vs. A smaller XXX,XXX pages on one site and XX,XXX pages on 10-20 other sites all pointing to the primary site. The thinking on the second was that those domains would likely achieve high DA as well as the primary, and would passing their value to the primary. Thoughts? Any other considerations we should be thinking about?
Intermediate & Advanced SEO | | intentionally0 -
Tips to Merge Two Domains?
Hi all, my client assigned me to merge both their two domains. To cut the long story, at the moment they have, for example, domain1.com and domain2.com. Practically, they sell quite the same products but with different content and layout. This has been happening for several years: domain1.com came first then domain2.com a couple of months after. They noticed that domain2.com has now more traffic than domain1.com. Another reason is they want to use one placeholder to avoid brand confusion (as expected because the products are actually the same). So, they are planning to merge the brand into domain2.com. I think this is just the same as moving domain, isn't it? I have been preparing all things as per SEOmoz SEO Guide: How to Properly Move Domains and Google's moving your site . Do you have any other tips to include? I want to make sure that the domain moving will be successful and the link juice on domain1.com can be transferred to domain2.com . Many thanks for your help! 🙂
Intermediate & Advanced SEO | | Ci3reR0 -
301 Redirecting multiple domains to brand new domain
Hi guys, I have read quite a bit of stuff on 301 redirects after Penguin. Hoping someone could help me out. im looking at a way to do a legit 301 redirect without passing the penalty. I have acquired two businesses, business1 and business2, that both had websites that were hit by penguin. Ive anaylsed there backlinks and theres a lot of spammy forum links and comments and I was also informed they were both using buildmyrank. A side note, buiness2 only started using BMR after it noticed business1 have large amounts of high PR links. business1.com was ranking at position 1 till the penguin hit. Business2.com was ranking around page 2 I work in the same arena as these two businesses and didnt generate any business via the internet. When these 2 businesses failed (due to loss of rankings and traffic) i decided to take them over. What I am thinking of doing is 301'ing both business domains to my brand new, zero links, domain which will be the name of my new company. I will combine the content from both sites, around 1000 pages, in to the new one. So my question is, does 301'ing multiple domains, that target the same keywords, and operate in the same niche, look less "spammy" then 301'ing 1 domain? I'm trying to look at it in the eyes of google. It is a legit merging of businesses. Thanks for your help, really appreciate your time
Intermediate & Advanced SEO | | JohnPeters0 -
Multiple domains expiring that have 301 redirects to my primary domain. Am I in trouble?
I recently took on the SEO of a large website with http://example.com. My predecessor bought 40 plus domains for specific cities like Jacksonvilleexample.com, Miamiexample.com, etc. ZERO of the additional domains linked to our main website. The domains that were bought basically had our exact same website in terms of content, links etc that mirrored our main http://example.com. I added 301 redirects to help problems that may be a result of this type of structure. Some of the additional domains were indexed and some were not but all have 301's and as far as traffic is concerned I'm not worried about loosing short term traffic. My question: All the domains are set to expire in June and I don't want to continue to have them 301 redirected to my main domain (example.com). I'm not trying to avoid the additional cost of all the domains but I don't see an advantage to having them so CAN letting all these domains expire hurt me from a long term SEO position if I don't renew them?
Intermediate & Advanced SEO | | ballanrk0 -
Content on New Domain or Sub Directory of Existing Domain?
I have a client with a well aged, high DA site. They rank well for their wedding photography business in several cities. They are launching a new service which is related to photography (photobooths and flipbooks) which they built and developed content on a new domain. The existing domain has 0 links with a DA of 1. The site is brand new.. Is there any drawback to moving the existing content on the new domain to a sub directory of the high authority domain? EX: http://domain.com/newcompany The look, feel, and design of the new site / service is much different than the high DA site. My thoughts are that this will give them an automatic step up, especially since they will be marketing this in several major cities. Also, since the design will be different, if it is good to move to the subdir, should we put the new company name in the subdir folder or something keyword friendly like domain.com/photobooth as opposed to domain.com/newcompanyname. Any thoughts would be greatly appreciated.
Intermediate & Advanced SEO | | itrogers0 -
Are sites that leave out www from domain at a disadvantage to domains with www in url
I know this has been discussed but was wondering what would be the best approach from an SEO perspective. I quite like the idea of setting up websites with domains without www but always worry that setting up domains without www has a disadvantage because user are use to referring to sites with the www included. Thus one of my fears are that users would link back using www version which will mean even if you do a 301 redirect that some of the link juice would be lost. I know some famous sites have used this convention such as http://searchenginewatch.com/ so think it would be possible but still concerned that for new sites it would be better to rather stick to conventions. What are your opinions about this?
Intermediate & Advanced SEO | | SABest0