Disallow: /404/ - Best Practice?
-
Hello Moz Community,
My developer has added this to my robots.txt file: Disallow: /404/
Is this considered good practice in the world of SEO? Would you do it with your clients?
I feel he has great development knowledge but isn't too well versed in SEO.
Thank you in advanced,
Nico.
-
Thank you Lesley.
This really helps a lot. I appreciate it very much. This is my site by the way: http://devilswink.com/
Thanks.
Nico.
-
This comes down to personal preference in my opinion. I think honestly it is neither here nor there. The instances that your 404 page could come up in the SERP's is more than likely pretty low and at the same time it really does not offer any useful content. So disallowing it would not really be any loss. One reason why it might be disallowed is that you have an e-commerce site that rotates products. When a product is deleted, the developer has a 301 to the 404 page, then with the robots.txt saying not to index the 404 page, the other page will drop out of search engines. If this is the case I would rethink that strategy. If you notice a lot of sites like amazon and other big sites leave the page in the index even if the product is no longer for sale. The thought is traffic is traffic, the hardest part in the whole equation is getting someone to your site, if the page is ranking, why delete it.
The only time I can think that I would specifically allow it and optimize it is if you have a cool 404 page. Some companies actually spend a bit of time on their pages and it gets them a little pop of viral traffic from social sharing sites like reddit. If you do have one that is funny or unique I would allow it and actually optimize it for a term like "funny 404 page" or something like that.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will this URL structure: "domain.com/s/content-title" cause problems?
Hey all, We have a new in-house built too for building content. The problem is it inserts a letter directly after the domain automatically. The content we build with these pages aren't all related, so we could end up with a bunch of urls like this: domain.com/s/some-calculator
Technical SEO | | joshuaboyd
domain.com/s/some-infographic
domain.com/s/some-long-form-blog-post
domain.com/s/some-product-page Could this cause any significant issues down the line?0 -
Duplicate content and 404 errors
I apologize in advance, but I am an SEO novice and my understanding of code is very limited. Moz has issued a lot (several hundred) of duplicate content and 404 error flags on the ecommerce site my company takes care of. For the duplicate content, some of the pages it says are duplicates don't even seem similar to me. additionally, a lot of them are static pages we embed images of size charts that we use as popups on item pages. it says these issues are high priority but how bad is this? Is this just an issue because if a page has similar content the engine spider won't know which one to index? also, what is the best way to handle these urls bringing back 404 errors? I should probably have a developer look at these issues but I wanted to ask the extremely knowledgeable Moz community before I do 🙂
Technical SEO | | AliMac260 -
Hey all -- ever seen a client with URLs that keep repeating the domain? Something like: client.com/client.com/client.com/subfolder-name. Any idea what glitch could cause that?
Hey all -- ever seen a client with URLs that keep repeating the domain? Something like: client.com/client.com/client.com/subfolder-name. Any idea what glitch could cause that?
Technical SEO | | TDC_SEO0 -
Removing links - Best practice
Hi I have noticed on webmaster that I have a lot of links to my sites from link building directories. Either I did this many years a go or somehow they've linked to me. Would links to link building directories harm my site? i.e linkspurt.com pingerati.net I have quite a few and just wondering what to do with them. Also I have some customer sites which are massive one site has 38,000 links coming to my site as I have put a credit that I built the site with a link back to mine. It has a low score in Google would this also harm my site? Any advise would be appreciated.
Technical SEO | | Cocoonfxmedia0 -
How best to deal with www.home.com and www.home.com/index.html
Firstly, this is for an .asp site - and all my usual ways of fixing this (e.g. via htaccess) don't seem to work. I'm working on a site which has www.home.com and www.home.com/index.html - both URL's resolve to the same page/content. If I simply drop a rel canonical into the page, will this solve my dupe content woes? The canonical tag would then appear in both www.home.com and www.home.com/index.html cases. If the above is Ok, which version should I be going with? - or - Thanks in advance folks,
Technical SEO | | Creatomatic
James @ Creatomatic0 -
Correct Redirect method for switching pages from .html to /pretty urls/
I have a customer that has all his site files as .html extensions and i'm going to rebuild this site into a wordpress site for easier management, regarding the new permalink structure, should i just do a 301 redirect on this?
Technical SEO | | tgr0ss0 -
Any idea why this is reporting a 404 in MozTools?
I did away with a vague category and 301 redirected the category url to the home page. However the link is reporting as a 404 in Moz Tools when it scans my site. Here's the link, and as you can see it redirects to the home page. Just curious if I did something wrong. Thanks.
Technical SEO | | NoahsDad0 -
Canonicalisation - Best Approach?
Hi I have been reading into canonicalisation and I am looking for professional advice and guidance on the best approach as there are many conflicting views on how to implement a solution. I have added to the homepage http://www.cheapfindergames.com/" /> so that http://cheapfindergames.com and index.php redirects to the www version. Or would it be better using .htaccess if this is the case what code would be needed to achieve the above? Many Thanks
Technical SEO | | ocelot0