undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • MozCon
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Digital Marketers
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    Track AI Overviews in Keyword Research
    Moz Pro

    Track AI Overviews in Keyword Research

    Try it free!
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Digital Marketers

      Simplify SEO tasks to save time and grow your traffic.

    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Intermediate & Advanced SEO
  4. Do search engines crawl links on 404 pages?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Do search engines crawl links on 404 pages?

Intermediate & Advanced SEO
3
8
2.2k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • brad-causes
    brad-causes last edited by Sep 17, 2013, 5:27 PM

    I'm currently in the process of redesigning my site's 404 page. I know there's all sorts of best practices from UX standpoint but what about search engines? Since these pages are roadblocks in the crawl process, I was wondering if there's a way to help the search engine continue its crawl.

    Does putting links to "recent posts" or something along those lines allow the bot to continue on its way or does the crawl stop at that point because the 404 HTTP status code is thrown in the header response?

    1 Reply Last reply Reply Quote 0
    • brad-causes
      brad-causes @loopyal last edited by Sep 18, 2013, 10:20 PM Sep 18, 2013, 10:20 PM

      Okay, thanks Alan!

      1 Reply Last reply Reply Quote 0
      • Matt-Williamson
        Matt-Williamson @brad-causes last edited by Sep 18, 2013, 5:17 AM Sep 18, 2013, 5:16 AM

        Hi Brad

        Sorry I have only just come back to you - it was late night here in the UK, but it looks like Alan has already answered your question 🙂

        Have you tested your 404 page with fetch as Google in webmaster tools - you should see that it can see the links on your 404 page and as such will continue crawling them as Alan has said.

        So what is a benefit to a user will also be a benefit to Google crawling your site in my opinion 🙂

        1 Reply Last reply Reply Quote 0
        • loopyal
          loopyal last edited by Sep 17, 2013, 9:26 PM Sep 17, 2013, 9:25 PM

          Sorry, yes, it should crawl the links - they used to do that.

          But you can prove it to yourself, by doing what I said - and then report back.

          brad-causes 1 Reply Last reply Sep 18, 2013, 10:20 PM Reply Quote 1
          • brad-causes
            brad-causes @loopyal last edited by Sep 17, 2013, 9:06 PM Sep 17, 2013, 9:06 PM

            Yes it will continue crawling or yes it will stop the crawl?

            1 Reply Last reply Reply Quote 0
            • loopyal
              loopyal last edited by Sep 17, 2013, 8:34 PM Sep 17, 2013, 8:34 PM

              Yes and you can test it by creating a page that is linked from nowhere else and then check your logs or analytics

              brad-causes 1 Reply Last reply Sep 17, 2013, 9:06 PM Reply Quote 0
              • brad-causes
                brad-causes @Matt-Williamson last edited by Sep 17, 2013, 7:26 PM Sep 17, 2013, 7:15 PM

                Hey Matt,

                Thanks for the reply. I'm aware of all the best practice stuff but thanks for sending through. It didn't quite answer my question so let me rephrase...

                Will a bot follow a hyperlink (like the example below) on a 404 page or will it stop the crawl on that page (not on the whole site) because the header response code is a 404?

                Recent Post Title

                Matt-Williamson 1 Reply Last reply Sep 18, 2013, 5:16 AM Reply Quote 0
                • Matt-Williamson
                  Matt-Williamson last edited by Sep 17, 2013, 6:10 PM Sep 17, 2013, 5:57 PM

                  Hi Brad

                  Firstly it is great from a usability point of view to have a custom 404 page and I would link it to your most popular content and maybe add a search feature on the page for your site to help find the content that is missing. I have come across some nice 404s that actually have very concise sitemap in order to help the visitor navigate the site.In order to prevent Google from indexing your 404 page you need to make sure it returns an actuall 404 HTTP status code.

                  In order to understand how Goolgebot crawls your site I would look at the following post from Google themselves - https://support.google.com/webmasters/answer/182072?hl=en

                  Rather than being concerned about a 404 page having links on to keep the crawl going make sure you have an XML sitemap that you have submitted to Google via Webmaster Tools as this will help your crawl process.

                  Googlebot alots a set amount of time to crawling your site and it doesn't just stop crawling because it encounters a 404 error. However make sure that you monitor Google Webmaster Tools and take care of any reported 404s with 301 redirects for instance if the page has changed location. You will notice that Googlebot reports 404 erros on the days it finds them and these can often be multiple 404 errors encountered in one visit to your site by Googlebot. Keeing an eye on this and making sure you keep it updated will make your site as crawl efficient as possible which is clearly what you are after - as we all are 😄

                  I thought this would also be interesting reading in relation to this - http://googlewebmastercentral.blogspot.co.uk/2011/05/do-404s-hurt-my-site.html

                  Hope this helps

                  brad-causes 1 Reply Last reply Sep 17, 2013, 7:15 PM Reply Quote 0
                  • 1 / 1
                  1 out of 8
                  • First post
                    1/8
                    Last post

                  Got a burning SEO question?

                  Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                  Start my free trial


                  Browse Questions

                  Explore more categories

                  • Moz Tools

                    Chat with the community about the Moz tools.

                  • SEO Tactics

                    Discuss the SEO process with fellow marketers

                  • Community

                    Discuss industry events, jobs, and news!

                  • Digital Marketing

                    Chat about tactics outside of SEO

                  • Research & Trends

                    Dive into research and trends in the search industry.

                  • Support

                    Connect on product support and feature requests.

                  • See all categories

                  Related Questions

                  • Nobody1616542228134

                    Is a page with links to all posts okay?

                    Hi folks. Instead of an archive page template in my theme (I have my reasons), I am thinking of simply typing the post title as and when I publish a post, and linking to the post from there. Any SEO issues that you can think of? Thanks in advance!

                    Intermediate & Advanced SEO | Mar 31, 2021, 7:27 AM | Nobody1616542228134
                    0
                  • DebFF

                    404 Errors flaring on nonexistent or unpublished pages – should we be concerned for SEO?

                    Hello! We keep getting "critical crawler" notifications on Moz because of firing 404 codes. We've checked each page and know that we are not linking to them anywhere on our site, they are not published and they are not indexed on Google. It's only happened since we migrated our blog to Hubspot so we think it has something to do with the test pages their developers had set up and that they are just lingering in our code somewhere. However, we are still concerned having these codes fire implies negative consequences for our SEO. Is this the case? Should we be concerned about these 404 codes despite the pages from those URLs not actually existing? Thank you!
                    Chloe

                    Intermediate & Advanced SEO | Sep 17, 2019, 5:55 AM | DebFF
                    0
                  • lohardiu9

                    Can you disallow links via Search Console?

                    Hey guys, Is it possible in anyway to nofollow links via search console (not disavow) but just nofollow external links pointing to your site? Cheers.

                    Intermediate & Advanced SEO | Oct 3, 2017, 1:14 PM | lohardiu9
                    0
                  • _nitman

                    What's the best possible URL structure for a local search engine?

                    Hi Mozzers, I'm working at AskMe.com which is a local search engine in India i.e if you're standing somewhere & looking for the pizza joints nearby, we pick your current location and share the list of pizza outlets nearby along with ratings, reviews etc. about these outlets. Right now, our URL structure looks like www.askme.com/delhi/pizza-outlets for the city specific category pages (here, "Delhi" is the city name and "Pizza Outlets" is the category) and www.askme.com/delhi/pizza-outlets/in/saket for a category page in a particular area (here "Saket") in a city. The URL looks a little different if you're searching for something which is not a category (or not mapped to a category, in which case we 301 redirect you to the category page), it looks like www.askme.com/delhi/search/pizza-huts/in/saket if you're searching for pizza huts in Saket, Delhi as "pizza huts" is neither a category nor its mapped to any category. We're also dealing in ads & deals along with our very own e-commerce brand AskMeBazaar.com to make the better user experience and one stop shop for our customers. Now, we're working on URL restructure project and my question to you all SEO rockstars is, what can be the best possible URL structure we can have? Assume, we have kick-ass developers who can manage any given URL structure at backend.

                    Intermediate & Advanced SEO | Aug 8, 2015, 1:46 AM | _nitman
                    0
                  • PeteC12

                    Ecommerce Site homepage , Is it okay to have Links as H2 Tags as that is relevant to the page ?

                    Hi All, I have a Rental site and I am bit confused with how best do my H Tags on my homepage I know the H1 is the most important, Then H2 Tags and so on.. and that these tags should really be titles for content. However, I have a few categories (links) on my homepage so I am wondering if I could put these as H2 Tags given that it is relevant to the page . H3 Tags will my News and Guides etc , H4 Tags will the whats on the footer. I am attached a made up screenshot of what I propose for my homepage  if someone could please  give it a quick look ,  it would be very much appreciated. I have looked at what some competitors do a lot of them don't seem to have h2's etc but I know it's an important factor for rankings etc. Many thanks Pete dJSFQwI

                    Intermediate & Advanced SEO | Oct 31, 2014, 1:33 PM | PeteC12
                    0
                  • AshShep1

                    Should my back links go to home page or internal pages

                    Right now we rank on page 2 for many KWs, so should i now focus my attention on getting links to my home page to build domain authority or continue to direct links to the internal pages for specific KWs? I am about to write some articles for several good ranking sites and want to know whether to link my company name (same as domain name) or KW to the home page or use individual KWs to the internal pages - I am only allowed one link per article to my site. Thanks Ash

                    Intermediate & Advanced SEO | Aug 7, 2014, 11:25 AM | AshShep1
                    0
                  • JenniferDacosta

                    Is it a bad idea to have a "press" page and link to press mentions of our company?

                    We've recently been getting quite a bit of press. Would it be wise to create a "press" page and link to mentions of us or would this devalue the links on the press pages as Google may think they reciprocal?

                    Intermediate & Advanced SEO | Mar 20, 2013, 10:01 PM | JenniferDacosta
                    0
                  • BeytzNet

                    How to properly link to products from category pages?

                    Hi All, We have an e-commerce website and the category pages are built so that there is a product image and below it there is the title. Both the image and the title are in a href (each on its own). I encountered the following unfinished discussion here at MOZ:
                    http://www.seomoz.org/q/how-to-optimize-achor-text-links-on-ecommerce-category-page#post-93758 The discussion states that its improper. The question is - if it is wrong then why? (maybe because Google will give its weight to the image anchor instead of the text anchor since it is higher in the page). The other question is how to resolve the matter?
                    Should I add nofollow to the image href? Thanks

                    Intermediate & Advanced SEO | Jun 17, 2012, 1:08 AM | BeytzNet
                    0

                  Get started with Moz Pro!

                  Unlock the power of advanced SEO tools and data-driven insights.

                  Start my free trial
                  Products
                  • Moz Pro
                  • Moz Local
                  • Moz API
                  • Moz Data
                  • STAT
                  • Product Updates
                  Moz Solutions
                  • SMB Solutions
                  • Agency Solutions
                  • Enterprise Solutions
                  • Digital Marketers
                  Free SEO Tools
                  • Domain Authority Checker
                  • Link Explorer
                  • Keyword Explorer
                  • Competitive Research
                  • Brand Authority Checker
                  • Local Citation Checker
                  • MozBar Extension
                  • MozCast
                  Resources
                  • Blog
                  • SEO Learning Center
                  • Help Hub
                  • Beginner's Guide to SEO
                  • How-to Guides
                  • Moz Academy
                  • API Docs
                  About Moz
                  • About
                  • Team
                  • Careers
                  • Contact
                  Why Moz
                  • Case Studies
                  • Testimonials
                  Get Involved
                  • Become an Affiliate
                  • MozCon
                  • Webinars
                  • Practical Marketer Series
                  • MozPod
                  Connect with us

                  Contact the Help team

                  Join our newsletter
                  Moz logo
                  © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                  • Accessibility
                  • Terms of Use
                  • Privacy

                  Looks like your connection to Moz was lost, please wait while we try to reconnect.