Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • Case Studies
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      What is your Brand Authority?
      Moz

      What is your Brand Authority?

      Check yours now
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • SEO Q&A

        Insights & discussions from an SEO community of 500,000+.

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • Case Studies

        Explore how Moz drives ROI with a proven track record of success.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Technical SEO
    4. What tool do you use to check for URLs not indexed?

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    What tool do you use to check for URLs not indexed?

    Technical SEO
    24
    25
    26190
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • elephantseo
      elephantseo last edited by

      What is your favorite tool for getting a report of URLs that are not cached/indexed in Google & Bing for an entire site? Basically I want a list of URLs not cached in Google and a seperate list for Bing.

      Thanks,

      Mark

      Shiv24k 1 Reply Last reply Reply Quote 3
      • austinbenz8282
        austinbenz8282 last edited by

        I've had good results using Google Search Console for checking which URLs are indexed. It's pretty straightforward and gives a clear overview of any indexing issues halloweensquishmallows.

        1 Reply Last reply Reply Quote 0
        • Shiv24k
          Shiv24k @elephantseo last edited by

          @elephantseo https://docs.google.com/document/d/e/2PACX-1vQprHYh4feCfnhC72ldSnbCG0OoXBgqpKHC2bvlbs7hX3XZ8axnrY-LuIDS5-BJ_0mSsUlcT0_OyUvp/pub

          1 Reply Last reply Reply Quote 0
          • noitcudni
            noitcudni last edited by

            I can work on building this tool if there's enough interest.

            1 Reply Last reply Reply Quote 0
            • Adlanera
              Adlanera last edited by

              I generally just use Xenu's hyperlink sleuth (if you thousands of pages) to listing out all the URLs you have got and I might then manually take a look at them, however, see the guitar in demand I have not come upon an automatic device yet. If all people are aware of any, I'd like to recognize as properly.

              1 Reply Last reply Reply Quote 0
              • CarlLarson
                CarlLarson last edited by

                This post from Distilled mentions that SEO for Excel plugin has a "Indexation Checker":
                https://www.distilled.net/blog/seo/awesome-examples-of-how-to-use-seotools-for-excel/

                Alas, after downloading and installing, it appears this feature was removed...

                1 Reply Last reply Reply Quote 0
                • CMC-SD
                  CMC-SD @kdaly100 last edited by

                  Unless I'm missing something, there doesn't seem to be a way to get Google to show more than 100 results on a page. Our site has about 8,000 pages, and I don't relish the idea of manually exporting 80 SERPs.

                  1 Reply Last reply Reply Quote 0
                  • rickv101
                    rickv101 last edited by

                    Annie Cushing from Seer Interactive made an awesome list of all the must have tools for SEO.

                    You can get it from her link which is http://bit.ly/tools-galore

                    In the list there is a tool called scrapebox which is great for this. In fact there are many uses for the software, it is also useful for sourcing potential link partners.

                    1 Reply Last reply Reply Quote 0
                    • Philip_Petrescu
                      Philip_Petrescu @Chris.McGiff last edited by

                      I would suggest using the Website Auditor from Advanced Web Ranking. It can parse 10.000 pages and it will tell you a lot more info than just if it's indexed by Google or not.

                      1 Reply Last reply Reply Quote 0
                      • squareplug
                        squareplug last edited by

                        hmm...I thought there was a way to pull those SERPs urls into Google docs using a function of some sort?

                        1 Reply Last reply Reply Quote 0
                        • DanishWadhwa
                          DanishWadhwa last edited by

                          I think you need not any tool for this, you can directly go to google.com and search: Site:www.YourWebsiteNem.com Site:www.YourWebsiteName.com/directory I think this will be the best option to check if your website is crwled by google or not.

                          1 Reply Last reply Reply Quote 0
                          • Chris.McGiff
                            Chris.McGiff @DavidKauzlaric last edited by

                            I do something similar but use Advanced Web Ranking, use site:www.domain.com as your phrase, run it to retrieve 1000 results and generate a Top Site Report in Excel to get the indexed list.

                            Also remember that you can do it on sub-directories (or partial URL paths) as a way to get more than 1000 pages from the site. In general I run it once with site:www.domain.com, then identify the most frequent sub-directories, and add those as additional phrases to the project and run a second time, i.e.: site:www.domain.com site:www.domain.com/dir1 site:www.domain.com/dir2 etc.

                            Still not definitive, but think it does give indication of where value is.

                            Philip_Petrescu 1 Reply Last reply Reply Quote 1
                            • lonniea
                              lonniea last edited by

                              David Kauzlaric has in my opinion the best answer. If google hasn't indexed it and you've investigated your Google webmaster account, then there isn't anything better out there as far as I'm concerned. It's by far the simplest, quickest and easiest way to identify a serp result.

                              re: David Kauzlaric

                              We built an internal tool to do it for us, but basically you can do this manually.

                              Go to google, type in "site:YOURURLHERE" without the quotes. You can check a certain page, a site, a subdomain, etc... of course if you have thousands of URLs this method is not ideal, but it can be done.

                              Cheers!

                              1 Reply Last reply Reply Quote -1
                              • ezclickmedia
                                ezclickmedia @Syed1 last edited by

                                I concur, Xenu is an extremely valuable tool for me that I use daily. Also, once you get a list of all the URLs on your site, you can compare the two lists in excel (two lists being the Xenu page list for your site and the list of pages that have been indexed by Google).

                                1 Reply Last reply Reply Quote 0
                                • ShaMenz
                                  ShaMenz @kdaly100 last edited by

                                  Nice solution Kieran!

                                  I use the same method, to compare URL list from Screaming Frog output with URL Found column from my Keyword Ranking tool - of course it doesn't catch all pages that might be indexed.

                                  The intention is not really to get a complete list, more to "draught" out pages that need work.

                                  1 Reply Last reply Reply Quote 0
                                  • NakulGoyal
                                    NakulGoyal @kdaly100 last edited by

                                    I agree, this is not automated but so far, from what we know, looks like a nice and clean option. Thanks.

                                    1 Reply Last reply Reply Quote 0
                                    • kdaly100
                                      kdaly100 last edited by

                                      Saw this and tried the following which isn't automated but is one way of doing it.

                                      • First install SEO Quake plugin
                                      • Go to Google
                                      • Turn off Google Instant (http://www.google.com/preferences)
                                      • Go to Advanced search set the number of results you want displayed (estimate the number of pages on your site)
                                      • Then run your site:www.example.com search query
                                      • Export this to CSV
                                      • Import to Excel
                                      • Once then do a Data to columns conversion using ; as a delimiter (this is the CSV delimiter)
                                      • This gives you a formatted list.
                                      • Then import your sitemap.xml into another TAB in Excel
                                      • Run a vlookup between the URL tabs to flag which are on sitemap or vice versa.

                                      Not exactly automated but does the job.

                                      NakulGoyal ShaMenz CMC-SD 3 Replies Last reply Reply Quote 15
                                      • Hakkasan
                                        Hakkasan last edited by

                                        Curious about this question also, it would be very useful to see a master list of all URLs on our site that are not indexed by Google so that we can take action to see what aspects of the page are lacking and what we need for it to get indexed.

                                        1 Reply Last reply Reply Quote 0
                                        • Syed1
                                          Syed1 last edited by

                                          I usually just use Xenu's link sleuth (if you thousands of pages) to list out all the URLs you have and I would then manually check them, but I haven't come across an automated tool yet. If anyone knows any, I'd love to know as well.

                                          ezclickmedia 1 Reply Last reply Reply Quote 1
                                          • VisualSense
                                            VisualSense Subscriber last edited by

                                            Manual is a no go for large sites. If someone knows a tool like this, it woul be cool to know which/ where to find. Or..... This would make a cool SEOmoz pro tool 🙂

                                            1 Reply Last reply Reply Quote 4
                                            • Function5
                                              Function5 @KeriMorgret last edited by

                                              My bad - you are right that it doesn't display the actual URLs. So I guess the best thing you can do is site:examplesite.com and see what comes up.

                                              1 Reply Last reply Reply Quote 0
                                              • KeriMorgret
                                                KeriMorgret @Function5 last edited by

                                                That will tell you the number indexed, but it still doesn't tell you which of those URLs are or are not indexed. I think we all wish it would!

                                                Function5 1 Reply Last reply Reply Quote 1
                                                • Function5
                                                  Function5 last edited by

                                                  I would use Google Webmaster Tools as you can see how many URLs are indexed based on your sitemap. Once you have that, you can compare it to your total list. The same can be done with Bing.

                                                  KeriMorgret 1 Reply Last reply Reply Quote 1
                                                  • elephantseo
                                                    elephantseo @DavidKauzlaric last edited by

                                                    Yeah I do it manually now so was looking for something more efficient.

                                                    1 Reply Last reply Reply Quote 0
                                                    • DavidKauzlaric
                                                      DavidKauzlaric last edited by

                                                      We built an internal tool to do it for us, but basically you can do this manually.

                                                      Go to google, type in "site:YOURURLHERE" without the quotes. You can check a certain page, a site, a subdomain, etc... of course if you have thousands of URLs this method is not ideal, but it can be done.

                                                      elephantseo Chris.McGiff 2 Replies Last reply Reply Quote 2
                                                      • 1 / 1
                                                      • First post
                                                        Last post

                                                      Got a burning SEO question?

                                                      Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                                                      Start my free trial


                                                      Browse Questions

                                                      Explore more categories

                                                      • Moz Tools

                                                        Chat with the community about the Moz tools.

                                                      • SEO Tactics

                                                        Discuss the SEO process with fellow marketers

                                                      • Community

                                                        Discuss industry events, jobs, and news!

                                                      • Digital Marketing

                                                        Chat about tactics outside of SEO

                                                      • Research & Trends

                                                        Dive into research and trends in the search industry.

                                                      • Support

                                                        Connect on product support and feature requests.

                                                      • See all categories

                                                      Related Questions

                                                      • ghesta

                                                        In writing the url, it is better to use the language used by the people of my country or English?

                                                        We speak Persian and all people search in Persian on Google. But I read in some sources that the url should be in English. Please tell me which language to use for url writing?
                                                        For example, I brought down two models: 1fb0e134-10dc-4737-904f-bfdf07143a98-image.png https://ghesta.ir/blog/how-to-become-rich/
                                                        2)https://ghesta.ir/blog/چگونه-پولدار-شویم/

                                                        Technical SEO | | ghesta
                                                        0
                                                      • muzzmoz

                                                        Indexed pages

                                                        Just started a site audit and trying to determine the number of pages on a client site and whether there are more pages being indexed than actually exist. I've used four tools and got four very different answers... Google Search Console: 237 indexed pages Google search using site command: 468 results MOZ site crawl: 1013 unique URLs Screaming Frog: 183 page titles, 187 URIs (note this is a free licence, but should cut off at 500) Can anyone shed any light on why they differ so much? And where lies the truth?

                                                        Technical SEO | | muzzmoz
                                                        1
                                                      • MonicaOConnor

                                                        No Index PDFs

                                                        Our products have about 4 PDFs a piece, which really inflates our indexed pages. I was wondering if I could add REL=No Index to the PDF's URL? All of the files are on a file server, so they are embedded with links on our product pages. I know I could add a No Follow attribute, but I was wondering if any one knew if the No Index would work the same or if that is even possible. Thanks!

                                                        Technical SEO | | MonicaOConnor
                                                        0
                                                      • Creditsafe

                                                        No index on subdomains

                                                        Hi, We have a subdomain that is appearing in the search results - I want to hide this as it looks really bad. If I were to add the no index tag to the sub domain would URL would this affect the whole domain or just that sub domain? The main domain is vitally important - it is just that sub domain I need to hide. Many thanks

                                                        Technical SEO | | Creditsafe
                                                        0
                                                      • UnderRugSwept

                                                        Google is indexing my directories

                                                        I'm sure this has been asked before, but I was looking at all of Google's results for my site and I found dozens of results for directories such as: Index of /scouting/blog/wp-includes/js/swfupload/plugins Obviously I don't want those indexed. How do I prevent Google from indexing those? Also, it only seems to be doing it with Wordpress, not any of the directories on my main site. (We have a wordpress blog, which is only a portion of the site)

                                                        Technical SEO | | UnderRugSwept
                                                        0
                                                      • waynekolenchuk

                                                        Instant Indexing

                                                        I've been  working on a site for a while now, methodically building content and building trust and authority. Lately I've noticed that anything I publish there appears to be instantly indexed by Google, which surprises me. I haven't had this happen before so I'm curious. I'd be interested to hear the experience of others.

                                                        Technical SEO | | waynekolenchuk
                                                        0
                                                      • kaje

                                                        Old URL redirect to New URL

                                                        Alright I did something dumb a year a go and I'm still paying for it. I changed my hyphenated URL to the non-hyphenated version when I redesigned my website. I say it was dumb because I lost most of my link juice even though I did 301 redirects (via the htaccess file) for almost all of the pages I could find in Google's index. Here's my problem. My new site took a huge hit in traffic (down 60%) when I made the change and even though I've done thousands of redirects my old site is still showing up in the SERPS and send much if not most of my traffic. I don't want to take the old site down in fear it will kill all of my traffic. What should I do? Is there a better method I should explore then 301 redirects? Could the other site be affecting my current rank since it's still there? (FYI...both sites are built on the WP platform). Any help or ideas are greatly appreciated. Thank you! Joe

                                                        Technical SEO | | kaje
                                                        0
                                                      • surveygizmo

                                                        Does Google pass link juice a page receives if the URL parameter specifies content and has the Crawl setting in Webmaster Tools set to NO?

                                                        The page in question receives a  lot of quality traffic but is only relevant to a small percent of my users. I want to keep the link juice received from this page but I do not want it to appear in the SERPs.

                                                        Technical SEO | | surveygizmo
                                                        0

                                                      Get started with Moz Pro!

                                                      Unlock the power of advanced SEO tools and data-driven insights.

                                                      Start my free trial
                                                      Products
                                                      • Moz Pro
                                                      • Moz Local
                                                      • Moz API
                                                      • Moz Data
                                                      • STAT
                                                      • Product Updates
                                                      Moz Solutions
                                                      • SMB Solutions
                                                      • Agency Solutions
                                                      • Enterprise Solutions
                                                      Free SEO Tools
                                                      • Domain Authority Checker
                                                      • Link Explorer
                                                      • Keyword Explorer
                                                      • Competitive Research
                                                      • Brand Authority Checker
                                                      • MozBar Extension
                                                      • MozCast
                                                      Resources
                                                      • Blog
                                                      • SEO Learning Center
                                                      • Help Hub
                                                      • Beginner's Guide to SEO
                                                      • How-to Guides
                                                      • Moz Academy
                                                      • API Docs
                                                      About Moz
                                                      • About
                                                      • Team
                                                      • Careers
                                                      • Contact
                                                      Why Moz
                                                      • Case Studies
                                                      • Testimonials
                                                      Get Involved
                                                      • Become an Affiliate
                                                      • MozCon
                                                      • Webinars
                                                      • Practical Marketer Series
                                                      • MozPod
                                                      Connect with us

                                                      Contact the Help team

                                                      Join our newsletter
                                                      Moz logo
                                                      © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                                                      • Accessibility
                                                      • Terms of Use
                                                      • Privacy

                                                      Looks like your connection to Moz was lost, please wait while we try to reconnect.