undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Technical SEO
  4. Mass 404 Checker?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Mass 404 Checker?

Technical SEO
6
11
18.2k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • thisisOllie
    thisisOllie last edited by Oct 25, 2012, 11:10 AM

    Hi all,

    I'm currently looking after a collection of old newspaper sites that have had various developments during their time. The problem is there are so many 404 pages all over the place and the sites are bleeding link juice everywhere so I'm looking for a tool where I can check a lot of URLs at once.

    For example from an OSE report I have done a random sampling of the target URLs and some of them 404 (eek!) but there are too many to check manually to know which ones are still live and which ones have 404'd or are redirecting. Is there a tool anyone uses for this or a way one of the SEOMoz tools can do this?

    Also I've asked a few people personally how to check this and they've suggested Xenu, Xenu won't work as it only checks current site navigation.

    Thanks in advance!

    1 Reply Last reply Reply Quote 0
    • keboozgur
      keboozgur last edited by Jan 21, 2017, 9:48 PM Jan 21, 2017, 9:48 PM

      Hi,

      we are seo agency at turkey, our name  clicksus. We can deadlinkchecker.com and it is very easy & good.

      1 Reply Last reply Reply Quote 0
      • topic:timeago_earlier,4 years
      • ThompsonPaul
        ThompsonPaul @thisisOllie last edited by Oct 26, 2012, 6:21 PM Oct 26, 2012, 6:21 PM

        Glad I was able to help!

        It would be great if you could mark the answers you found helpful, and mark the question as answered if you feel you got the information you needed. That will make it even more useful for other users.

        Paul

        1 Reply Last reply Reply Quote 0
        • thisisOllie
          thisisOllie @ThompsonPaul last edited by Oct 26, 2012, 5:30 AM Oct 26, 2012, 5:30 AM

          Wow nice one mate did not know that in the Top Pages tab that is perfect! I'll remember to click around more often now.

          I found this tool on my adventures which was exactly what I was after: http://www.tomanthony.co.uk/tools/bulk-http-header-compare/

          Also cheers for your walkthrough, having problems with the site still bleeding 404 pages, first thing first however is fixing these pages getting high quality links to them 🙂

          Cheers again!

          ThompsonPaul 1 Reply Last reply Oct 26, 2012, 6:21 PM Reply Quote 0
          • ThompsonPaul
            ThompsonPaul last edited by Oct 30, 2012, 2:42 PM Oct 25, 2012, 2:04 PM

            Sorry, one additional - since you mentioned using Open Site Explorer...

            Go to the Top Pages tab in OSE and filter the results to include only incoming links. One of the columns in that report is HTTP Status. It will tell you if the linked page's status is 404. Again, just download the full CSV, sort the resulting spreadsheet by the Status column and you'll be able to generate a list of URLs that no longer have pages associated with them to start fixing.

            Paul

            thisisOllie 1 Reply Last reply Oct 26, 2012, 5:30 AM Reply Quote 1
            • ThompsonPaul
              ThompsonPaul last edited by Oct 30, 2012, 2:42 PM Oct 25, 2012, 1:54 PM

              Ollie, if I'm understanding your question correctly, the easiest place for you to start is with Google Webmaster Tools. You're looking to discover URLs of pages that used to exist on the sites, but no longer do, yes?

              If you click on the Health link in left sidebar, then click Crawl Errors, you get a page showing different kinds of errors the Google crawler has detected. Click on the Not Found error box and you'll get a complete list of all the pages Google is aware of that can no longer be found on your site (i.e. 404s).

              You can then download the whole list as a CSV and start cleaning them up from there.

              This list will basically include pages that have been linked to at one time or another from other sites on the web, so while not exhaustive, it will show the ones that are most likely to still be getting traffic. For really high-value incoming links, you might even want to contact the linking site and see if you can get them to relink to the correct new page.

              Alternatively, if you can access the sites' server logs, they will record all the incoming 404s with their associated URLs as well and you can get a dump from the log files to begin creating your work list. I just find it's usually easier to get access to Webmaster Tools than to get at a clients server log files.

              Is that what you're looking for?

              Paul

              1 Reply Last reply Reply Quote 1
              • Andy.Drinkwater
                Andy.Drinkwater @MarkScully last edited by Oct 25, 2012, 11:51 AM Oct 25, 2012, 11:51 AM

                To be honest, I don't know anyone who has bad things to say about Screaming Frog - aside from the cost, but as you said, really worth it.

                However, it is free for up to 500 page crawl limit, so perhaps give it a go?

                Andy

                1 Reply Last reply Reply Quote 0
                • thisisOllie
                  thisisOllie last edited by Oct 25, 2012, 11:47 AM Oct 25, 2012, 11:47 AM

                  Cheers Andy & Kyle

                  Problem with this tool as it works similar to Xenu which is great for making sure your current navigation isn't causing problems.

                  My problem is there are over 15k links pointing to all sorts of articles and I have no idea what's live and what's not.  Running the site through that tool won't report the pages that aren't linked in the navigation anymore but are still being linked to.

                  Example is manually checking some of the links I've found that the site has quite a few links from the BBC going to 404 pages.  Running the site through Xenu or Screamy Frog doesn't find these pages.

                  Ideally I'm after a tool I can slap in a load of URLs and it'll do a simple HTTP header check on them.  Only tools I can find do 1 or 10 at a time which would take quite a while trying to do 15k!

                  1 Reply Last reply Reply Quote 0
                  • MarkScully
                    MarkScully last edited by Oct 25, 2012, 11:45 AM Oct 25, 2012, 11:45 AM

                    Agree with Screaming Frog. It's more comprehensive than **Xenu's Link Sleuth. **

                    It costs £99 for a year but totally worth it.

                    I had a few issues with Xenu taking too long to compile a report or simply crashing.

                    Andy.Drinkwater 1 Reply Last reply Oct 25, 2012, 11:51 AM Reply Quote 1
                    • kyleNeedham
                      kyleNeedham last edited by Oct 25, 2012, 11:39 AM Oct 25, 2012, 11:39 AM

                      Xenu Liunk Seuth - its free and will go through internal links, external or both, it will also show you where the 404 page is being linked from.

                      Also can report 302s.

                      http://home.snafu.de/tilman/xenulink.html

                      1 Reply Last reply Reply Quote 0
                      • Andy.Drinkwater
                        Andy.Drinkwater last edited by Oct 25, 2012, 11:32 AM Oct 25, 2012, 11:32 AM

                        Screaming Frog Spider does a pretty good job...

                        As simple as enter the URL and leave it to report back when completed.

                        Andy

                        1 Reply Last reply Reply Quote 0
                        • 1 / 1
                        1 out of 11
                        • First post
                          1/11
                          Last post

                        Got a burning SEO question?

                        Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                        Start my free trial


                        Browse Questions

                        Explore more categories

                        • Moz Tools

                          Chat with the community about the Moz tools.

                        • SEO Tactics

                          Discuss the SEO process with fellow marketers

                        • Community

                          Discuss industry events, jobs, and news!

                        • Digital Marketing

                          Chat about tactics outside of SEO

                        • Research & Trends

                          Dive into research and trends in the search industry.

                        • Support

                          Connect on product support and feature requests.

                        • See all categories

                        Related Questions

                        • Mediaholix

                          Does anyone know the linking of hashtags on Wix sites does it negatively or postively impact SEO. It is coming up as an error in site crawls 'Pages with 404 errors' Anyone got any experience please?

                          Does anyone know the linking of hashtags on Wix sites does it negatively or positively impact SEO.  It is coming up as an error in site crawls 'Pages with 404 errors'  Anyone got any experience please? For example at the bottom of this blog post https://www.poppyandperle.com/post/face-painting-a-global-language the hashtags are linked, but they don't go to a page, they go to search results of all other blogs using that hashtag.  Seems a bit of a strange approach to me.

                          Technical SEO | Feb 24, 2020, 1:23 PM | Mediaholix
                          0
                        • ReSEOlve

                          404 errors

                          Hi I am getting these show up in WMT crawl error any help would be very much appreciated | ?escaped_fragment=Meditation-find-peace-within/csso/55991bd90cf2efdf74ec3f60 | 404 | 12/5/15 |
                          | | 2 | mobile/?escaped_fragment= | 404 | 10/26/15 |
                          | | 3 | ?escaped_fragment=Tips-for-a-balanced-lifestyle/csso/1 | 404 | 12/1/15 |
                          | | 4 | ?escaped_fragment=My-favorite-yoga-spot/csso/5598e2130cf2585ebcde3b9a | 404 | 12/1/15 |
                          | | 5 | ?escaped_fragment=blog/c19s6 | 404 | 11/29/15 |
                          | | 6 | ?escaped_fragment=blog/c19s6/Tag/yoga | 404 | 11/30/15 |
                          | | 7 | ?escaped_fragment=Inhale-exhale-and-once-again/csso/2 | 404 | 11/27/15 |
                          | | 8 | ?escaped_fragment=classes/covl | 404 | 10/29/15 |
                          | | 9 | m/?escaped_fragment= | 404 | 10/26/15 |
                          | | 10 | ?escaped_fragment=blog/c19s6/Page/1 | 404 | 11/30/15 | |   |

                          Technical SEO | Dec 10, 2015, 7:01 PM | ReSEOlve
                          0
                        • SEO_Promenade

                          Best strategy to handle over 100,000 404 errors.

                          I recently been given a site that has over one-hundred thousand 404 error codes listed in Google Webmasters. It is really odd because according to Google Webmasters, the pages that are linking to these 404 pages are also pages that no longer exist (they are 404 pages themselves). These errors were a result of site migration that had occurred. Appreciate any input on how one might go about auditing and repairing large amounts of 404 errors. Thank you.

                          Technical SEO | Aug 15, 2014, 2:48 PM | SEO_Promenade
                          0
                        • Taswirh

                          Expired domain 404 crawl error

                          I recently purchased a Expired domain from auction and after I started my new site on it, I am noticing 500+ "not found" errors in Google Webmaster Tools, which are generating from the previous owner's contents.Should I use a redirection plugin to redirect those non-exist posts to any new post(s) of my site? or I should use a 301 redirect? or I should leave them just as it is without taking further action? Please advise.

                          Technical SEO | Jul 19, 2014, 8:29 PM | Taswirh
                          1
                        • Oxfordcomma

                          Increase 404 errors or 301 redirects?

                          Hi all, I'm working on an e-commerce site that sells products that may only be available for a certain period of time. Eg. A product may only be selling for 1 year and then be permanently out of stock. When a product goes out of stock, the page is removed from the site regardless of any links it may have gotten over time. I am trying to figure out the best way to handle these permanently out of stock pages. At the moment, the site is set up to return a 404 page for each of these products. There are currently 600 (and increasing) instances of this appearing on Google Webmasters. I have read that too many 404 errors may have a negative impact on your site, and so thought I might 301 redirect these URLs to a more appropriate page. However I've also read that too many 301 redirects may have a negative impact on your site. I foresee this to be an issue several years down the road when the site has thousands of expired products which will result in thousands of 404 errors or 301 redirects depending on which route I take. Which would be the better route? Is there a better solution?

                          Technical SEO | Jun 6, 2013, 5:08 AM | Oxfordcomma
                          0
                        • Prime85

                          Are 404 Errors a bad thing?

                          Good Morning... I am trying to clean up my e-commerce site and i created a lot of new categories for my parts... I've made the old category pages (which have had their content removed) "hidden" to anyone who visits the site and starts browsing.  The only way you could get to those "hidden" pages is either by knowing the URLS that I used to use or if for some reason one of them is spidering in Google. Since I'm trying to clean up the site and get rid of any duplicate content issues, would i be better served by adding those "hidden" pages that don't have much or any content to the Robots.txt file or should i just De-activate them so now even if you type the old URL you will get a 404 page... In this case, are 404 pages bad?  You're typically not going to find those pages in the SERPS so the only way you'd land on these 404 pages is to know the old url i was using that has been disabled. Please let me know if you guys think i should be 404'ing them or adding them to Robots.txt Thanks

                          Technical SEO | Jan 20, 2013, 9:53 PM | Prime85
                          0
                        • WEB-IRS

                          No Search Results Found - Should this return status code 404?

                          A question came up today on how to correctly serve the right status code on pages where no search results are found. I did a couple searches on some major eccomerce and news sites and they were ALL serving status code 200 for No Search Results Found http://www.zappos.com/dsfasdgasdgadsg http://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Daps&field-keywords=sdafasdklgjasdklgjsjdjkl http://www.ebay.com/sch/i.html?_trksid=p5197.m570.l1313&_nkw=dfjakljgdkslagklasd&_sacat=0 http://www.cnn.com/search/?query=sdgadgdsagas&x=0&y=0&primaryType=mixed&sortBy=date&intl=false http://www.seomoz.org/pages/search_results?q=sdagasdgasdgasg I thought I read somewhere were it was recommended to serve a status code 404 on these types of pages.  Based on what I found above, all sites were serving a 200, so it appears this may not be the best practice. Any thoughts?

                          Technical SEO | Jun 20, 2012, 10:36 PM | WEB-IRS
                          0
                        • EugeneF

                          404 crawl errors from "tel:" link?

                          I am seeing thousands of 404 errors.  Each of the urls is like this: abc.com/abc123/tel:1231231234 Everything is normal about that url except the "/tel:1231231234" these urls are bad with the tel: extension, they are good without it. The only place I can find this character string is on each page we have this code which is used for Iphones and such.  What are we doing wrong? Code: Phone: <a href="[tel:1231231234](tel:7858411943)"> (123) 123-1234a>

                          Technical SEO | Feb 20, 2012, 8:01 PM | EugeneF
                          0

                        Get started with Moz Pro!

                        Unlock the power of advanced SEO tools and data-driven insights.

                        Start my free trial
                        Products
                        • Moz Pro
                        • Moz Local
                        • Moz API
                        • Moz Data
                        • STAT
                        • Product Updates
                        Moz Solutions
                        • SMB Solutions
                        • Agency Solutions
                        • Enterprise Solutions
                        Free SEO Tools
                        • Domain Authority Checker
                        • Link Explorer
                        • Keyword Explorer
                        • Competitive Research
                        • Brand Authority Checker
                        • Local Citation Checker
                        • MozBar Extension
                        • MozCast
                        Resources
                        • Blog
                        • SEO Learning Center
                        • Help Hub
                        • Beginner's Guide to SEO
                        • How-to Guides
                        • Moz Academy
                        • API Docs
                        About Moz
                        • About
                        • Team
                        • Careers
                        • Contact
                        Why Moz
                        • Case Studies
                        • Testimonials
                        Get Involved
                        • Become an Affiliate
                        • MozCon
                        • Webinars
                        • Practical Marketer Series
                        • MozPod
                        Connect with us

                        Contact the Help team

                        Join our newsletter
                        Moz logo
                        © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                        • Accessibility
                        • Terms of Use
                        • Privacy

                        Looks like your connection to Moz was lost, please wait while we try to reconnect.