How Best to Handle Inherited 404s on Purchased Domain
-
We purchased a domain from another company and migrated our site over to it very successfully. However, we have one artifact of the original domain in that there was a page that was exploited by other sites on the web. This page allowed you to pass any URL to it and redirect to that URL (e.g. http://example.com/go/to/offsite_link.asp?GoURL=http://badactor.com/explicit_content).
This page does not exist on our site so the results always go to a 404 on our site. However, we find that crawlers are still attempting to access these invalid pages.
We have disavowed as many of the explicit sites as we can, but still some crawlers come looking for those links. We are considering blocking the redirect page in our robots.txt but we are concerned that the links will remain indexed but uncrawlable.
What's the best way to pull these pages from search engines and never have them crawled again?
UPDATE: Clarifying that what we're trying to do it get search engines to just never try to get to these pages. We feel the fact they're even wasting their time on getting a 404 is what we're trying to avoid. Is there any reason we shouldn't just block these in our robots.txt?
-
@gastonriera calm down mate. We have actually tested this at not seen any negative effect on any site we have done it on. It is the "easiest" option, but it won't cause the death and destruction your comment implies. Good day sir.
-
Hi there,
I'm considering that you have over 500k URLs, to be worrying about crawl efficiency. If you have less than that, please don't worry.
Having 404s is completely fine, and google will eventually lower their crawl frequency to those pages.
Blocking them in robots.txt will cause to google stop crawling them, but never to never remove them from the index.
My advice here: don't block them in robots.txtAs Rajesh pointed out, you could force those 404s into 410 to tell Google that they are gone forever. Yet, Google said that they treat 404s and 410s as the same.
John Mueller said over a year ago that 4xx status codes don't incur in crawl wastage. You can check it our in these Webmasters hangout notes - DeepcrawlHope it helps,
Best luck.
Gaston -
FOR THE LOVE OF GOD DONT REDIRECT 404s TO THE HOME!
This is terrible advice. Doing that you'll turn those 404s into soft 404s, making them more problematic than ever.
-
I would actually recommend redirecting it to the homepage. If you have a Wordpress website and a bunch of 404 pages, you can install a free plugin called "All 404 to Homepage" and this will solve the problem. I would, however, recommend that if you have replacement pages or pages covering similar content, that you redirect those to the corresponding replacement page.
-
You need to do one thing with those 404 pages. Move them as 410 status code. Redirection is not good practice for the same.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Merger & Acquisition Best Practices
Our company (DA 40) recently acquired another company (DA 20). The domain for the acquired company is up, and I am being asked if we should keep paying for the domain. I have a couple of questions regarding this. Is it best practice to continue paying for the older domain and do 301 redirects to our website? If yes, do I have to add 301 redirects for individual matching pages? For example, if the old site has the topic XYZ, should a 301 go to our site if we have a similar page topic XYZ? What about contact pages, about us, etc? Do we redirect all non-matching topics to the new home page? Or in the case of their blogs, do we redirect their blogs to the new blog home page even though we are not keeping their old blogs? What if our company keeps acquiring other companies, are we to assume we have to keep paying for the domains of the acquired companies? Is there anyone out there that would say stop paying for the old domain and just review the company's inbound links, reach out to those sites, and make sure they link to the new site instead? Thanks for the help in advance.
Intermediate & Advanced SEO | | CharityHBS0 -
Is .ME domain is effective in SEO ?
I am always listening about TLD. com. org .net but what about the .me domain. Can this will be effective in SEO. Can i able to beat down my competitors, if i choose .me . I also have a .com or other TLD option but if i am making my name than .me is for me but i need your suggestion for the seo purpose. Is there really domain affective in term of SEO.
Intermediate & Advanced SEO | | pnb5670 -
Can we really learn from the best?
Hi All, When I started my site (an eCommerce site) I copied (or tried) a lot of things from the best eCommerce sites I thought were out there. Sites like Zappos, ZALES, Overstock, BlueNile etc. I got hit pretty hard with latest algo changes and I posted my question at Google Webmaster Help forum I received answers from Gurus that we are keyword stuffing etc. (mainly with internal links to product pages but other issues as well). My answer was a link to Zappos and other sites showing that what we do is nothing compared to them. I also showed dozens of SEO "errors" like using H1 tag 10 times per page, not using canonicals and many other issues. The Guru's answer was "LOL" - who am I to compare myself to Zappos. So the question is... Can we take them for example or are they first simply because they are the biggest?
Intermediate & Advanced SEO | | BeytzNet0 -
Duplicate Sub-domains Being Indexed
Hi all, I have this site that has a sub-domain that is meant to be a "support" for clients. Some sort of FAQ pages, if you will. A lot of them are dynamic URLs, hence, the title and most of the content are duplicated. Crawl Diagnostics found 52 duplicate content, 138 duplicate title and a lot other errors. My question is, what would be the best practice to fix this issue? Should I noindex and nofollow all of its subdomains? Thanks in advance.
Intermediate & Advanced SEO | | EdwardDennis0 -
Multi domain redirect to single domain
Hello, all SEOers. Today, I would like to get some ideas about handling multiple domains. I have a client who bought numerous domains under purpose of prevent abuse of their brand name and at the same time for future uses. This client bought more than 100 domains. Some domains are paused, parked, lived and redirected to other site. I don't worry too much of parked domains and paused domains. However, what I am worrying is that there are about 40 different domains are now redirected to single domain and meta refresh was used for redirections. As far as I know, this can raise red flag for Google. I asked clients to clean up unnecessary domains, yet they want to keep them all. So now I have to figure out how to handle all domains which are redirect to single domain. So far, I came up with following ideas. 1. Build gateway page which shows lists of my client sites and redirect all domains to gateway page. 2. Implement robots.txt file to all different domains 3. Delete the redirects and leave it as parked domains. Could anyone can share other ideas in order to handling current status? Please people, share your ideas for me.
Intermediate & Advanced SEO | | Artience0 -
Keywords in domains losing power
One of my clients received a letter from another seo company touting for business. I would be interested in hearing what your opinion is on this fellow SEO providers. Letter attached - > letter.jpg
Intermediate & Advanced SEO | | onlinemediadirect0 -
7 years old domain sandboxed for 8 months, wait or make a domain change?
Hello folks The questions is, if a domain, 7 years old being sandboxed due to "notice of unnatural links to website" does it make sense to make a domain change (301 permanent redirect and make a "domain change" under google webmaster tools) to another, aged(!) domain name?
Intermediate & Advanced SEO | | Ferray
Website being sandboxed for over 8 months already and there is no chance to do anything with those "unnatural" links to website... Any suggestions?0 -
Exact domain or subfolder?
If I am targeting a specific keyword, from an SEO perspective is it better to create a subfolder on a url that has some authority or is it better use the exact domain with no authority? For example, if I want to target the word 'widgets' which is the better choice and why? **Choice 1: ** www.domainwithauthority.com/widgets Note: this domain has 1000 links to it **Choice 2: ** www.widgets.com Note: this is a brand new domain with 0 links
Intermediate & Advanced SEO | | mnipko0