Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • MozCon
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Technical SEO
    4. Exclude status codes in Screaming Frog

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Exclude status codes in Screaming Frog

    Technical SEO
    3
    8
    2544
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • DonnaDuncan
      DonnaDuncan last edited by

      I have a very large ecommerce site I'm trying to spider using screaming frog. Problem is I keep hanging even though I have turned off the high memory safeguard under configuration.

      The site has approximately 190,000 pages according to the results of a Google site: command.

      • The site architecture is almost completely flat. Limiting the search by depth is a possiblity, but it will take quite a bit of manual labor as there are literally hundreds of directories one level below the root.
      • There are many, many duplicate pages. I've been able to exclude some of them from being crawled using the exclude configuration parameters.
      • There are thousands of redirects. I haven't been able to exclude those from the spider b/c they don't have a distinguishing character string in their URLs.

      Does anyone know how to exclude files using status codes? I know that would help.

      If it helps, the site is kodylighting.com.

      Thanks in advance for any guidance you can provide.

      1 Reply Last reply Reply Quote 0
      • CHAD215
        CHAD215 last edited by

        Thanks for your help. It literally was just the fact that it had to be done before the crawl began and could not be changed during the crawl. Hopefully this is changed because sometimes during a crawl you find things you want to exclude that you may have not known of their existence before hand.

        1 Reply Last reply Reply Quote 0
        • MickEdwards
          MickEdwards @CHAD215 last edited by

          Are you sure it's just on Mac,have you tried on PC?   Do you have any other rules in include or perhaps a conflicting rule in exclude? Try running a single exclude rule, also on another small site to test.

          Also from support if failing on all fronts:

          • Mac version, please make sure you have the most up to date version of the OS which will update Java.
          • Please uninstall, then reinstall the spider ensuring you are using the latest version and try again.

          To be sure - http://www.youtube.com/watch?v=eOQ1DC0CBNs

          1 Reply Last reply Reply Quote 0
          • CHAD215
            CHAD215 last edited by

            does the exclude function work on mac. i have tried every possible way to exclude folders and have not been successful while running an analysis

            MickEdwards 1 Reply Last reply Reply Quote 0
            • DonnaDuncan
              DonnaDuncan @MickEdwards last edited by

              That's exactly the problem, the redirects are disbursed randomly throughout the site. Although, and the job's still running, it now appears as though there's almost a 1-2-1 correlation between pages and redirects on the site.

              I also heard from Dan Sharp via Twitter. He said "You can't, as we'd have to crawl a URL to see the status code 😉 You can right click and remove after though!"

              Thanks again Michael. Your thoroughness and follow through is appreciated.

              1 Reply Last reply Reply Quote 0
              • MickEdwards
                MickEdwards @DonnaDuncan last edited by

                Took another look, also looked at documentation/online and don't see any way to exclude URLs from crawl based on response codes.  As I see it you would only want to exclude on name or directory as response code is likely to be random throughout a site and impede a thorough crawl.

                DonnaDuncan 1 Reply Last reply Reply Quote 1
                • DonnaDuncan
                  DonnaDuncan @MickEdwards last edited by

                  Thank you Michael.

                  You're right. I was on a 64 bit machine running a 32 bit verson of java. I updated it and the scan has been running for more than 24 hours now without hanging. So thank you.

                  If anyone else knows of a way to exclude files using status codes I'd still like to learn about it. So far the scan is showing me 20,000 redirected files which I'd just as soon not inventory.

                  MickEdwards 1 Reply Last reply Reply Quote 0
                  • MickEdwards
                    MickEdwards last edited by

                    I don't think you can filter out on response codes.

                    However, first I would ensure you are running the right version of Java if you are on a 64bit machine.  The 32bit version functions but you cannot increase the memory allocation which is why you could be running into problems.  Take a look at http://www.screamingfrog.co.uk/seo-spider/user-guide/general/ under Memory.

                    DonnaDuncan 1 Reply Last reply Reply Quote 1
                    • 1 / 1
                    • First post
                      Last post

                    Got a burning SEO question?

                    Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                    Start my free trial


                    Browse Questions

                    Explore more categories

                    • Moz Tools

                      Chat with the community about the Moz tools.

                    • SEO Tactics

                      Discuss the SEO process with fellow marketers

                    • Community

                      Discuss industry events, jobs, and news!

                    • Digital Marketing

                      Chat about tactics outside of SEO

                    • Research & Trends

                      Dive into research and trends in the search industry.

                    • Support

                      Connect on product support and feature requests.

                    • See all categories

                    Related Questions

                    • BakeryTech

                      Unsolved Using NoIndex Tag instead of 410 Gone Code on Discontinued products?

                      ecommerce noindex shopify indexed urls

                      Hello everyone, I am very new to SEO and I wanted to get some input & second opinions on a workaround I am planning to implement on our Shopify store. Any suggestions, thoughts, or insight you have are welcome & appreciated! For those who aren't aware, Shopify as a platform doesn't allow us to send a 410 Gone Code/Error under any circumstance. When you delete or archive a product/page, it becomes unavailable on the storefront. Unfortunately, the only thing Shopify natively allows me to do is set up a 301 redirect. So when we are forced to discontinue a product, customers currently get a 404 error when trying to go to that old URL. My planned workaround is to automatically detect when a product has been discontinued and add the NoIndex meta tag to the product page. The product page will stay up but be unavailable for purchase. I am also adjusting the LD+JSON to list the products availability as Discontinued instead of InStock/OutOfStock.
                      Then I let the page sit for a few months so that crawlers have a chance to recrawl and remove the page from their indexes. I think that is how that works?
                      Once 3 or 6 months have passed, I plan on archiving the product followed by setting up a 301 redirect pointing to our internal search results page. The redirect will send the to search with a query aimed towards similar products. That should prevent people with open tabs, bookmarks and direct links to that page from receiving a 404 error. I do have Google Search Console setup and integrated with our site, but manually telling google to remove a page obviously only impacts their index. Will this work the way I think it will?
                      Will search engines remove the page from their indexes if I add the NoIndex meta tag after they have already been index?
                      Is there a better way I should implement this? P.S. For those wondering why I am not disallowing the page URL to the Robots.txt, Shopify won't allow me to call collection or product data from within the template that assembles the Robots.txt. So I can't automatically add product URLs to the list.

                      Technical SEO | | BakeryTech
                      0
                    • KyleSennikoff

                      Unsolved Question about a Screaming Frog crawling issue

                      technical seo crawl errors

                      Hello, I have a very peculiar question about an issue I'm having when working on a website. It's a WordPress site and I'm using a generic plug in for title and meta updates. When I go to crawl the site through screaming frog, however, there seems to be a hard coded title tag that I can't find anywhere and the plug in updates don't get crawled. If anyone has any suggestions, thatd be great. Thanks!

                      Technical SEO | | KyleSennikoff
                      0
                    • AfroSEO

                      422 vs 404 Status Codes

                      We work with an automotive industry platform provider and whenever a vehicle is removed from inventory, a 404 error is returned. Being that inventory moves so quickly, we have a host of 404 errors in search console. The fix that the platform provider proposed was to return a 422 status code vs a 404. I'm not familiar with how a 422 may impact our optimization efforts. Is this a good approach, since there is no scalable way to 301 redirect all of those dead inventory pages.

                      Technical SEO | | AfroSEO
                      0
                    • vmsolu

                      How to redirect 302 status to 301 status code using wordpress

                      I just ran the link opportunity option within site explorer and it shows that 31 pages are currently in a 302 status.  Should I try to convert the 302's to 301's?  And what is the easiest way to do this? I see several wordpress plugins that claim to do 301 redirects but I don't know which to choose. Any help would be greatly appreciated!

                      Technical SEO | | vmsolu
                      0
                    • EcommerceSite

                      Do I have a problem with missing pages in Screaming Frog?

                      We have category pages and some of those pages have pagination due to us having additional items. Screaming Frog could not find the items that were after page 1. Is this a problem for Google? These item pages are still in the sitemap. I am sure they can find them to index them but does it hurt rankings at all.

                      Technical SEO | | EcommerceSite
                      0
                    • Nightwing

                      Screaming From occurences and canonicals what does it all mean

                      Bonjourno from Wetherby UK... Ive used a package called screamong frog to diagnose canonical errors but can anyone tell me what this means? http://i216.photobucket.com/albums/cc53/zymurgy_bucket/understand-occurances-canonical.jpg Thanks in advance. David

                      Technical SEO | | Nightwing
                      0
                    • Wallander

                      Should WordPress themes be hard coded for better SEO?

                      In the interests of making my site faster I have recently come across the suggestion of removing unwanted PHP from my WooThemes WordPress theme. The suggestion is to hard code the choices I have made in the WordPress template to reduce on database calls. Has anyone actually done this to their WordPress theme before and seen any measurable results?

                      Technical SEO | | Wallander
                      1
                    • ARMofficial

                      Is there such thing as a good text/code ratio? Can it effect SERPs?

                      As it says on the tin; Is there such thing as a good text/code ratio? And can it effect SERPs? I'm currently looking at a 20% ratio whereas some competitors are closer to 40%+. Best regards,
                      Sam.

                      Technical SEO | | ARMofficial
                      0

                    Get started with Moz Pro!

                    Unlock the power of advanced SEO tools and data-driven insights.

                    Start my free trial
                    Products
                    • Moz Pro
                    • Moz Local
                    • Moz API
                    • Moz Data
                    • STAT
                    • Product Updates
                    Moz Solutions
                    • SMB Solutions
                    • Agency Solutions
                    • Enterprise Solutions
                    Free SEO Tools
                    • Domain Authority Checker
                    • Link Explorer
                    • Keyword Explorer
                    • Competitive Research
                    • Brand Authority Checker
                    • Local Citation Checker
                    • MozBar Extension
                    • MozCast
                    Resources
                    • Blog
                    • SEO Learning Center
                    • Help Hub
                    • Beginner's Guide to SEO
                    • How-to Guides
                    • Moz Academy
                    • API Docs
                    About Moz
                    • About
                    • Team
                    • Careers
                    • Contact
                    Why Moz
                    • Case Studies
                    • Testimonials
                    Get Involved
                    • Become an Affiliate
                    • MozCon
                    • Webinars
                    • Practical Marketer Series
                    • MozPod
                    Connect with us

                    Contact the Help team

                    Join our newsletter
                    Moz logo
                    © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                    • Accessibility
                    • Terms of Use
                    • Privacy

                    Looks like your connection to Moz was lost, please wait while we try to reconnect.