Do you add 404 page into robot file or just add no index tag?
-
Hi,
got different opinion on this so i wanted to double check with your comment is.
We've got /404.html page and I was wondering if you would add this page to robot text so it wouldn't be indexed or would you just add no index tag? What would be the best approach?
Thanks!
-
Hello Rubix,
Saijo gave you some great advice, but I'm concerned about the fact that you have that page in the first place, and that it produces those URL parameters. It suggests to me that instead of showing a 404 error on the contact-office.aspx page (assuming that pages doesn't exist on that URL) you are redirecting the user who tries to access that URL to the /404.html page (e.g. /404.html?aspxerrorpath=/contact-office.aspx).
Typically you want the 404 http status code to show on the URL the user is trying to unsuccessfully access. In this case instead of redirecting them to your "404 page URL" you would want to show your customized 404 message (and ensure it returns a 404 status code, use this tool) on www.yourdomain.com/contact-office.aspx.
I hope this makes sense to you. If not, feel free to ask for clarification.
-
404 are OK on your site just make sure you send the proper 404 header response for the 404 page ... Google does NOT index 404 pages ( as long as it sends the 404 header response ) , so you don't need to block them via robots.txt or meta robots.
Infact GWT warns you about these if they are able to crawl the so called 404 pages that doesn't send a 404 header response , so I think its a good idea NOT to noindex them you will get the warning if something is wrong.
Google will only index your 404 if you don't do that..they call it soft 404 : https://support.google.com/webmasters/answer/181708?hl=en
worth reading : http://moz.com/learn/seo/http-status-codes
-
Thanks Martijn,
I actually want to know what would you do for the 404 page itself. It is something like:
www.mainurl.com/404.html and for some reason this started to create some other links such as
www.mainrul.com/404.html?aspxerrorpath=/contact-office.aspx
Do you think I should add 404 page and subpages to Robot.txt ?
Thanks!
-
Hi Sida,
I would add a noindex to the page and as you also will return the 404 status code this is enough data for Google to tell not to index the page itself.
Hope this answers your question.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google slow to index pages
Hi We've recently had a product launch for one of our clients. Historically speaking Google has been quick to respond, i.e when the page for the product goes live it's indexed and performing for branded terms within 10 minutes (without 'Fetch and Render'). This time however, we found that it took Google over an hour to index the pages. we found initially that press coverage ranked until we were indexed. Nothing major had changed in terms of the page structure, content, internal linking etc; these were brand new pages, with new product content. Has anyone ever experienced Google having an 'off' day or being uncharacteristically slow with indexing? We do have a few ideas what could have caused this, but we were interested to see if anyone else had experienced this sort of change in Google's behaviour, either recently or previously? Thanks.
Intermediate & Advanced SEO | | punchseo0 -
Does >70 character title tag affect a pages ranking in search?
We are a publication that puts out hundreds of articles a month. We have +5000 medium priority errors showing that our title element tags are too long. The title tag is structured like this: [Headine] | [Publication Name that is 23 characters] . However, since we are a publication, it's not practical for us to try to limit the length of our title tags to 70 characters or less because doing so would make the titles of our content seem very unnatural. We also don't want to remove the branding because we want it to go with the article when it's shared (and to appear when some titles are short enough to allow room in SERPs). I understand the reasons for limiting characters to 70 or less with regard to SERP friendliness. We try to keep key phrases in the front. People are more likely to click on a page if they know what it's about etc etc. My question is, do the longer titles affect the ability for the page to rank in search? To put it a different way, if we altered all the +5000 of the title tags to fit within 70 characters, would the page authorities and our site's domain authority increase? I'd like to avoid needed to clean up 5000 pages if the medium priority errors aren't really hurting us. Any input is appreciated. Thanks!
Intermediate & Advanced SEO | | CatBrain1 -
Should Schema.org Tags go on every page?
Happy Monday Moz World! I am just wondering what are some best practices when using Schema.org Tags. For Example, I have a client who provides multiple services and provides unique content on each webpage. The design of each of the webpagesare unique, and conveys information differently. My question is: If each page of a company's website has unique content that describes a service or product, could I essentially change the url & description of the Schema Tag so that each of my pages are indexable by relationship to that page's content? Thanks ahead of time for the great responses! B/R Will
Intermediate & Advanced SEO | | MarketingChimp100 -
Adding hreflang tags - better on each page, or the site map?
Hello, I am wondering if there seems to be a preference for adding hreflang tags (from this article). My client just changed their site from gTLDs to ccTLDs, and a few sites have taken a pretty big traffic hit. One issue is definitely the amount of redirects to the page, but I am also going to work with the developer to add hreflang tags. My question is - is it better to add them to the header of each page, or the site map, or both, or something else? Any other thoughts are appreciated. Our Australia site, which was at least findable using Australia Google before this relaunch, is not showing up, even when you search the company name directly. Thanks!Lauryn
Intermediate & Advanced SEO | | john_marketade0 -
Canonical Tags being indexed on paginated results?
On a website I'm working on which has a search feature with paginated results, all of the pages of the search results are set with a canonical tag back to the first page of the search results, however Google is indexing certain random pages within the result set. I can literally do a search in Google and find a deep page in the results, click on it and view source on that page and see that it has a canonical tag leading back to the first page of the set. Has anyone experienced this before? Why would Google not honor a canonical tag if it is set correctly? I've seen several SEO techniques for dealing with pagination, is there another solution that you all recommend?
Intermediate & Advanced SEO | | IrvCo_Interactive0 -
PR Dilution and Number of Pages Indexed
Hi Mozzers, My client is really pushing for me to get thousands, if not millions of pages indexed through the use of long-tail keywords. I know that I can probably get quite a few of them into Google, but will this dilute the PR on my site? These pages would be worthwhile in that if anyone actually visits them, there is a solid chance they will convert to a lead do to the nature of the long-tail keywords. My suggestion is to run all the keywords for these thousands of pages through adwords to check the number of queries and only create pages for the ones which actually receive searches. What do you guys think? I know that the content needs to have value and can't be scraped/low-quality and pulling these pages out of my butt won't end well, but I need solid evidence to make a case either for or against it to my clients.
Intermediate & Advanced SEO | | Travis-W0 -
How do I get my Golf Tee Times pages to index?
I understand that Google does not want to index other search results pages, but we have a large amount of discount tee times that you can search for and they are displayed as helpful listing pages, not search results. Here is an example: http://www.activegolf.com/search-northern-california-tee-times?Date=8%2F21%2F2013&datePicker=8%2F21%2F2013&loc=San+Diego%2C+CA&coupon=&zipCode=&search= These pages are updated daily with the newest tee times. We don't exactly want every URL with every parameter indexed, but at least http://www.activegolf.com/search-northern-california-tee-times. It's weird because all of the tee times are viewable in the HTML and are not javascript. An example of similar pages would be Yelp, for example this page is indexed just fine - http://www.yelp.com/search?cflt=dogwalkers&find_loc=Lancaster%2C+MA I know ActiveGolf.com is not as powerful as Yelp but it's still strange that none of our tee times search pages are being indexed. Would appreciate any ideas out there!
Intermediate & Advanced SEO | | CAndrew14.0