Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Generating 404 Errors but the Pages Exist
-
Hey
I have recently come across an issue with several of a sites urls being seen as a 404 by bots such as Xenu, SEOMoz, Google Web Tools etc. The funny thing is, the pages exist and display fine.
This happens on many of the pages which use the Modx CMS, but the index is fine. The wordpress blog in /blog/ all works fine.
The only thing I can think of is that I have a conflict in the htaccess, but troubleshooting this is difficult, any tool I have found online seem useless.
Have tried to rollback to previous versions but still does not work.
Anyone had any experience of similar issues?
Many thanks
K.
-
FYI, we finally found our error. The short URL turned out to be the same name as the folder (photo-gallery) so once this was changed, wordpress was able to access the correct path. A bit of custom javascript had to be amended as well, but that was limited to our custom code. Using your web-sniffer.net link we were able to test immediately and fix it fairly quickly. Thank you for your help!
-
That's true Ryan I guess it is coding related really.
Issues like this are a real pain in the ass. And most people don't even check WMT to realise the issues exist. TBH, I don't check as often as I should.
-
I agree with you Paul.
As you pointed out one possible cause is a CMS-related issue which I would refer to as "coding" meaning something in the code which was used to present the website. Perhaps there is a better way to phrase it but nothing comes to mind at the moment.
Another possibility you mentioned is Litespeed which would be a server-side issue directly. Either way, it is a legitimate issue which should be addressed.
-
FWIW, I don't think it's a coding issue. If it were coding, it would either show a 200OK or it would show a 404. It wouldn't sometimes serve a 404.
If you're using Litespeed, I'd guarantee that is the issue and if you're using Joomla, it's another prime culprit.
-
Please keep in mind, that 404 error does not mean the page doesn't exist. It means your server, is sending a response code to indicate that it doesn't exist.
When I installed Litespeed on my server, this issue happened over and over again.
I believe Joomla for example, has some kind of security module that serves a 404 if a single IP requests a page too many times. I remember running SEOFrog on a friends Joomla site and tons of 404's were showing up.
-
Dev team are looking into it, must be quite a complex htaccess issue. Will get to the bottom of it this week and post any findings.
-
Thanks Ryan! I will get it looked at...Sue
-
@DentalID, the same reply I offered to Guy applies for you as well. This is an SEO issue which does need to be fixed. Something on your end is causing the page to show with a 403 response code. You really need a programmer to get in there and determine the root cause of the issue. You could try asking your web host if you have managed hosting, but this level of assistance would normally be outside the support of managed hosting.
-
Guy,
In looking at the page this appears to be a legitimate problem. Your server settings allow you to present a page with any header code you wish. You can 301 a page but still present the page with a 200 code if you want. Presently it appears the page is being presented fine but your server is offering a 404 header code.
I can't tell the actual source of the problem other then to say it appears to be on your end and should be fixed. I originally looked at the code with the MOZbar but then checked independently with another tool as well. http://web-sniffer.net/
All tools show a 404 header code for the page. This response code is generated by your web server.
-
We are having a similar problem with this URL: http://dentalimplantsportland.com/photo-gallery/ and also the following locations:
http://cosmeticdentistportland.net/photo-gallery/
http://dentalveneersportland.com/photo-gallery/
SEO Moz and Google webmaster tools show it as a 403 error but the pages display fine. I am not able to tell if this is really a problem for SEO or if we should reconstruct this gallery system and would really love your input.
This is Wordpress with a Spry gallery...
Thanks so much!
-
It is just a small affiliate site I am looking at - this page creates a 404.
http://www.insure-uk.com/post-office-car-insurance.html
Currently testing on some beta servers. Hopefully should fix soon as otherwise it will lose indexation.
-
I also see this now and again, but next crawl they fix themselfs. i assume robots can not always reach page for a number of reasons
-
Can you offer an example of a URL which is causing this problem?
-
I have had the same issues, I think it is often the bot's problem
Just to be certain check your links are correct and manually test them. Also ensure your sitemap is up to date and that you are not blocking the crawlers with metarobots, robots.txt, or some weird stuff in htaccess.
I have found that renaming pages or moving them will often cause 404 issues with crawlers
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I apply Canonical Links from my Landing Pages to Core Website Pages?
I am working on an SEO project for the website: https://wave.com.au/ There are some core website pages, which we want to target for organic traffic, like this one: https://wave.com.au/doctors/medical-specialties/anaesthetist-jobs/ Then we have basically have another version that is set up as a landing page and used for CPC campaigns. https://wave.com.au/anaesthetists/ Essentially, my question is should I apply canonical links from the landing page versions to the core website pages (especially if I know they are only utilising them for CPC campaigns) so as to push link equity/juice across? Here is the GA data from January 1 - April 30, 2019 (Behavior > Site Content > All Pages😞
Intermediate & Advanced SEO | | Wavelength_International0 -
Fresh page versus old page climbing up the rankings.
Hello, I have noticed that if publishe a webpage that google has never seen it ranks right away and usually in a descend position to start with (not great but descend). Usually top 30 to 50 and then over the months it slowly climbs up the rankings. However, if my page has been existing for let's say 3 years and I make changes to it, it takes much longer to climb up the rankings Has someone noticed that too ? and why is that ?
Intermediate & Advanced SEO | | seoanalytics0 -
What if page exists for desktop but not mobile?
I have a domain (no subdomains) that serves up different dynamic content for mobile/desktop pages--each having the exact same page url, kind of a semi responsive design, and will be using "Vary: User-Agent" to give Google a heads up on this setup. However, some of the pages are only valid for mobile or only valid for desktop. In the case of when a page is valid only for mobile (call it mysite.com/mobile-page-only ), Google Webmaster Tools is giving me a soft 404 error under Desktop, saying that the page does not exist, Apparently it is doing that because my program is actually redirecting the user/crawler to the home page. It appears from the info about soft 404 errors that Google is saying since it "doesn't exist" I should give the user a 404 page--which I can make it customized and give the user an option to go to the home page, or choose links from a menu, etc.. My concern is that if I tell the desktop bot that mysite.com/mobile-page-only basically is a 404 error (ie doesn't exist), that it could mess up the mobile bot indexing for that page--since it definitely DOES exist for mobile users.. Does anyone here know for sure that Google will index a page for mobile that is a 404 not found for desktop and vice versa? Obviously it is important to not remove something from an index in which it belongs, so whether Google is careful to differential the two is a very important issue. Has anybody here dealt with this or seen anything from Google that addresses it? Might one be better off leaving it as a soft 404 error? EDIT: also, what about Bing and Yahoo? Can we assume they will handle it the same way? EDIT: closely related question--in a case like mine does Google need a separate sitemap for the valid mobile pages and valid desktop pages even though most links will be in both? I can't tell from reading several q&a on this. Thanks, Ted
Intermediate & Advanced SEO | | friendoffood0 -
Images Returning 404 Error Codes. 301 Redirects?
We're working with a site that has gone through a lot of changes over the years - ownership, complete site redesigns, different platforms, etc. - and we are finding that there are both a lot of pages and individual images that are returning 404 error codes in the Moz crawls. We're doing 301 redirects for the pages, but what would the best course of action be for the images? The images obviously don't exist on the site anymore and are therefore returning the 404 error codes. Should we do a 301 redirect to another similar image that is on the site now or redirect the images to an actual page? Or is there another solution that I'm not considering (besides doing nothing)? We'll go through the site to make sure that there aren't any pages within the site that are still linking to those images, which is probably where the 404 errors are coming from. Based on feedback below it sounds like once we do that, leaving them alone is a good option.
Intermediate & Advanced SEO | | garrettkite0 -
Do 404 Pages from Broken Links Still Pass Link Equity?
Hi everyone, I've searched the Q&A section, and also Google, for about the past hour and couldn't find a clear answer on this. When inbound links point to a page that no longer exists, thus producing a 404 Error Page, is link equity/domain authority lost? We are migrating a large eCommerce website and have hundreds of pages with little to no traffic that have legacy 301 redirects pointing to their URLs. I'm trying to decide how necessary it is to keep these redirects. I'm not concerned about the page authority of the pages with little traffic...I'm concerned about overall domain authority of the site since that certainly plays a role in how the site ranks overall in Google (especially pages with no links pointing to them...perfect example is Amazon...thousands of pages with no external links that rank #1 in Google for their product name). Anyone have a clear answer? Thanks!
Intermediate & Advanced SEO | | M_D_Golden_Peak0 -
Why does my home page show up in search results instead of my target page for a specific keyword?
I am using Wordpress and am targeting a specific keyword..and am using Yoast SEO if that question comes up.. and I am at 100% as far as what they recommend for on page optimization. The target html page is a "POST" and not a "Page" using Wordpress definitions. Also, I am using this Pinterest style theme here http://pinclone.net/demo/ - which makes the post a sort of "pop-up" - but I started with a different theme and the results below were always the case..so I don't know if that is a factor or not. (I promise .. this is not a clever spammy attempt to promote their theme - in fact parts of it don't even work for me yet so I would not recommend it just yet...) I DO show up on the first page for my keyword.. however.. instead of Google showing the page www.mywebsite.com/this-is-my-targeted-keyword-page.htm Google shows www.mywebsite.com in the results instead. The problem being - if the traffic goes only to my home page.. they will be less likely to stay if they dont find what they want immediately and have to search for it.. Any suggestions would be appreciated!
Intermediate & Advanced SEO | | chunkyvittles0 -
Dynamic pages - ecommerce product pages
Hi guys, Before I dive into my question, let me give you some background.. I manage an ecommerce site and we're got thousands of product pages. The pages contain dynamic blocks and information in these blocks are fed by another system. So in a nutshell, our product team enters the data in a software and boom, the information is generated in these page blocks. But that's not all, these pages then redirect to a duplicate version with a custom URL. This is cached and this is what the end user sees. This was done to speed up load, rather than the system generate a dynamic page on the fly, the cache page is loaded and the user sees it super fast. Another benefit happened as well, after going live with the cached pages, they started getting indexed and ranking in Google. The problem is that, the redirect to the duplicate cached page isn't a permanent one, it's a meta refresh, a 302 that happens in a second. So yeah, I've got 302s kicking about. The development team can set up 301 but then there won't be any caching, pages will just load dynamically. Google records pages that are cached but does it cache a dynamic page though? Without a cached page, I'm wondering if I would drop in traffic. The view source might just show a list of dynamic blocks, no content! How would you tackle this? I've already setup canonical tags on the cached pages but removing cache.. Thanks
Intermediate & Advanced SEO | | Bio-RadAbs0 -
Soft 404's from pages blocked by robots.txt -- cause for concern?
We're seeing soft 404 errors appear in our google webmaster tools section on pages that are blocked by robots.txt (our search result pages). Should we be concerned? Is there anything we can do about this?
Intermediate & Advanced SEO | | nicole.healthline4