Disallow: /404/ - Best Practice?
-
Hello Moz Community,
My developer has added this to my robots.txt file: Disallow: /404/
Is this considered good practice in the world of SEO? Would you do it with your clients?
I feel he has great development knowledge but isn't too well versed in SEO.
Thank you in advanced,
Nico.
-
Thank you Lesley.
This really helps a lot. I appreciate it very much. This is my site by the way: http://devilswink.com/
Thanks.
Nico.
-
This comes down to personal preference in my opinion. I think honestly it is neither here nor there. The instances that your 404 page could come up in the SERP's is more than likely pretty low and at the same time it really does not offer any useful content. So disallowing it would not really be any loss. One reason why it might be disallowed is that you have an e-commerce site that rotates products. When a product is deleted, the developer has a 301 to the 404 page, then with the robots.txt saying not to index the 404 page, the other page will drop out of search engines. If this is the case I would rethink that strategy. If you notice a lot of sites like amazon and other big sites leave the page in the index even if the product is no longer for sale. The thought is traffic is traffic, the hardest part in the whole equation is getting someone to your site, if the page is ranking, why delete it.
The only time I can think that I would specifically allow it and optimize it is if you have a cool 404 page. Some companies actually spend a bit of time on their pages and it gets them a little pop of viral traffic from social sharing sites like reddit. If you do have one that is funny or unique I would allow it and actually optimize it for a term like "funny 404 page" or something like that.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best Practice for www and non www
How is the best way to handle all the different variations of a website in terms of www | non www | http | https? In Google Search Console, I have all 4 versions and I have selected a preference. In Open Site Explorer I can see that the www and non www versions are treated differently with one group of links pointing to each version of the same page. This gives a different PA score. eg. http://mydomain.com DA 25 PA 35 http://www.mydomain.com DA 19 PA 21 Each version of the home page having it's only set of links and scores. Should I try and "consolidate" all the scores into one page? Should I set up redirects to my preferred version of the website? Thanks in advance
Technical SEO | | I.AM.Strategist0 -
Best use of an old domain?
I've discovered that my clients website used to have another domain name, which they still own but don't use. It's doing OK considering its not been used for a few years - almost 6,000 backlinks showing on Majestic. So what's the best way of using this for SEO? I'm presuming some kind of redirecting? A simple redirect of everything on the domain to the new domain index page? Or going trough all the old pages and redirecting them one by one?
Technical SEO | | abisti20 -
Should I disallow crawl of my Job board?
MOZ crawler is telling me we have loads of duplicate content issues. We use a Job Board plugin on our Wordpress site and we have allot of duplicate or very similar jobs (usually just a different location), but the plugin doesn't allow us to add any rel canonical tags to the individual jobs. Should I disallow the /jobs/ url in the robots.txt file? This will solve the duplicate content issue but then Google wont be able to crawl any of the individual job listings Has anyone had any experience working with a job board plugin on Wordpress and had a similar issue, or can advise on how best to solve our duplicate content?? Thanks 🙂
Technical SEO | | O2C0 -
My 404 page is returning a 404
Hi there, Moz has highlighted that my 404 page is returning a 404... Looking at webmaster tools within crawl errors, it's the same story. The only big change on the website is that we recently moved to https for the entire site, so all pages have a 301 to the corresponding https page, including the old 404 http page. I don't know if that makes any difference? Any help or advice on how I reasolve this will be much appreciated. Thanks, Stuart
Technical SEO | | Stuart260 -
Best Place to Redirect 301 to?
Hey Everyone! I have an old site with hundreds of blog posts that are very spammy (duplicate content, keyword stuffed, and just plain bad content). I am going to redirect them and delete them from WordPress but I'm wondering where is the best place to redirect them to? Home page, other posts, other pages...? Any thoughts would be appreciated! Thanks!
Technical SEO | | adamxj21 -
404 Error from site - is this normal?
I have been trying to clean up any 404 errors. We keep getting the following: URL /include/vdimgck.php referring domain http://www.856d.c@m/plus/feedback.php rendered domain unclickable by adding the "@" since I do not know if it is safe. I just turned off the trackbacks and pings in the blog since I saw it was producing duplicate content and from what I read it is not worth keeping those with Wordpress. is vdimgck.php anything some here instantly recognizes ? It tops all our 404 errors, seems like a lot of requests. Thanks!
Technical SEO | | Force70 -
Best practice: unique meta descriptions on blog 'tag' pages
Hi everyone, I'm curious, are there best practices for introducing unique meta descriptions on blog tag pages (I'm using wordpress)? For instance, using platinum seo, on an original post, the meta description is either the excerpt or a specified custom sentence. It doesn't appear that platinum seo allows for custom descriptions on tag pages. Love to hear your thoughts. Thanks! Peter
Technical SEO | | peterdbaron1 -
Http:// vs http://www.
Why is it that when I run an "On Page Optimization Keyword Report" for my website I get a different score when using http://www.tandmkitchens.com vs http://tandmkitchens.com. My keyword is "Kitchen Remodeling" http://www.tandmkitchens.com scores an A http://tandmkitchens.com scores a B It's the same page yet one url scores higher than the other. Any help! Thanks
Technical SEO | | fun52dig
Gary0