Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • Case Studies
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      What is your Brand Authority?
      Moz

      What is your Brand Authority?

      Check yours now
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • SEO Q&A

        Insights & discussions from an SEO community of 500,000+.

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • Case Studies

        Explore how Moz drives ROI with a proven track record of success.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Technical SEO
    4. What tool do you use to check for URLs not indexed?

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    What tool do you use to check for URLs not indexed?

    Technical SEO
    24
    25
    26190
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • elephantseo
      elephantseo last edited by

      What is your favorite tool for getting a report of URLs that are not cached/indexed in Google & Bing for an entire site? Basically I want a list of URLs not cached in Google and a seperate list for Bing.

      Thanks,

      Mark

      Shiv24k 1 Reply Last reply Reply Quote 3
      • austinbenz8282
        austinbenz8282 last edited by

        I've had good results using Google Search Console for checking which URLs are indexed. It's pretty straightforward and gives a clear overview of any indexing issues halloweensquishmallows.

        1 Reply Last reply Reply Quote 0
        • Shiv24k
          Shiv24k @elephantseo last edited by

          @elephantseo https://docs.google.com/document/d/e/2PACX-1vQprHYh4feCfnhC72ldSnbCG0OoXBgqpKHC2bvlbs7hX3XZ8axnrY-LuIDS5-BJ_0mSsUlcT0_OyUvp/pub

          1 Reply Last reply Reply Quote 0
          • noitcudni
            noitcudni last edited by

            I can work on building this tool if there's enough interest.

            1 Reply Last reply Reply Quote 0
            • Adlanera
              Adlanera last edited by

              I generally just use Xenu's hyperlink sleuth (if you thousands of pages) to listing out all the URLs you have got and I might then manually take a look at them, however, see the guitar in demand I have not come upon an automatic device yet. If all people are aware of any, I'd like to recognize as properly.

              1 Reply Last reply Reply Quote 0
              • CarlLarson
                CarlLarson last edited by

                This post from Distilled mentions that SEO for Excel plugin has a "Indexation Checker":
                https://www.distilled.net/blog/seo/awesome-examples-of-how-to-use-seotools-for-excel/

                Alas, after downloading and installing, it appears this feature was removed...

                1 Reply Last reply Reply Quote 0
                • CMC-SD
                  CMC-SD @kdaly100 last edited by

                  Unless I'm missing something, there doesn't seem to be a way to get Google to show more than 100 results on a page. Our site has about 8,000 pages, and I don't relish the idea of manually exporting 80 SERPs.

                  1 Reply Last reply Reply Quote 0
                  • rickv101
                    rickv101 last edited by

                    Annie Cushing from Seer Interactive made an awesome list of all the must have tools for SEO.

                    You can get it from her link which is http://bit.ly/tools-galore

                    In the list there is a tool called scrapebox which is great for this. In fact there are many uses for the software, it is also useful for sourcing potential link partners.

                    1 Reply Last reply Reply Quote 0
                    • Philip_Petrescu
                      Philip_Petrescu @Chris.McGiff last edited by

                      I would suggest using the Website Auditor from Advanced Web Ranking. It can parse 10.000 pages and it will tell you a lot more info than just if it's indexed by Google or not.

                      1 Reply Last reply Reply Quote 0
                      • squareplug
                        squareplug last edited by

                        hmm...I thought there was a way to pull those SERPs urls into Google docs using a function of some sort?

                        1 Reply Last reply Reply Quote 0
                        • DanishWadhwa
                          DanishWadhwa last edited by

                          I think you need not any tool for this, you can directly go to google.com and search: Site:www.YourWebsiteNem.com Site:www.YourWebsiteName.com/directory I think this will be the best option to check if your website is crwled by google or not.

                          1 Reply Last reply Reply Quote 0
                          • Chris.McGiff
                            Chris.McGiff @DavidKauzlaric last edited by

                            I do something similar but use Advanced Web Ranking, use site:www.domain.com as your phrase, run it to retrieve 1000 results and generate a Top Site Report in Excel to get the indexed list.

                            Also remember that you can do it on sub-directories (or partial URL paths) as a way to get more than 1000 pages from the site. In general I run it once with site:www.domain.com, then identify the most frequent sub-directories, and add those as additional phrases to the project and run a second time, i.e.: site:www.domain.com site:www.domain.com/dir1 site:www.domain.com/dir2 etc.

                            Still not definitive, but think it does give indication of where value is.

                            Philip_Petrescu 1 Reply Last reply Reply Quote 1
                            • lonniea
                              lonniea last edited by

                              David Kauzlaric has in my opinion the best answer. If google hasn't indexed it and you've investigated your Google webmaster account, then there isn't anything better out there as far as I'm concerned. It's by far the simplest, quickest and easiest way to identify a serp result.

                              re: David Kauzlaric

                              We built an internal tool to do it for us, but basically you can do this manually.

                              Go to google, type in "site:YOURURLHERE" without the quotes. You can check a certain page, a site, a subdomain, etc... of course if you have thousands of URLs this method is not ideal, but it can be done.

                              Cheers!

                              1 Reply Last reply Reply Quote -1
                              • ezclickmedia
                                ezclickmedia @Syed1 last edited by

                                I concur, Xenu is an extremely valuable tool for me that I use daily. Also, once you get a list of all the URLs on your site, you can compare the two lists in excel (two lists being the Xenu page list for your site and the list of pages that have been indexed by Google).

                                1 Reply Last reply Reply Quote 0
                                • ShaMenz
                                  ShaMenz @kdaly100 last edited by

                                  Nice solution Kieran!

                                  I use the same method, to compare URL list from Screaming Frog output with URL Found column from my Keyword Ranking tool - of course it doesn't catch all pages that might be indexed.

                                  The intention is not really to get a complete list, more to "draught" out pages that need work.

                                  1 Reply Last reply Reply Quote 0
                                  • NakulGoyal
                                    NakulGoyal @kdaly100 last edited by

                                    I agree, this is not automated but so far, from what we know, looks like a nice and clean option. Thanks.

                                    1 Reply Last reply Reply Quote 0
                                    • kdaly100
                                      kdaly100 last edited by

                                      Saw this and tried the following which isn't automated but is one way of doing it.

                                      • First install SEO Quake plugin
                                      • Go to Google
                                      • Turn off Google Instant (http://www.google.com/preferences)
                                      • Go to Advanced search set the number of results you want displayed (estimate the number of pages on your site)
                                      • Then run your site:www.example.com search query
                                      • Export this to CSV
                                      • Import to Excel
                                      • Once then do a Data to columns conversion using ; as a delimiter (this is the CSV delimiter)
                                      • This gives you a formatted list.
                                      • Then import your sitemap.xml into another TAB in Excel
                                      • Run a vlookup between the URL tabs to flag which are on sitemap or vice versa.

                                      Not exactly automated but does the job.

                                      NakulGoyal ShaMenz CMC-SD 3 Replies Last reply Reply Quote 15
                                      • Hakkasan
                                        Hakkasan last edited by

                                        Curious about this question also, it would be very useful to see a master list of all URLs on our site that are not indexed by Google so that we can take action to see what aspects of the page are lacking and what we need for it to get indexed.

                                        1 Reply Last reply Reply Quote 0
                                        • Syed1
                                          Syed1 last edited by

                                          I usually just use Xenu's link sleuth (if you thousands of pages) to list out all the URLs you have and I would then manually check them, but I haven't come across an automated tool yet. If anyone knows any, I'd love to know as well.

                                          ezclickmedia 1 Reply Last reply Reply Quote 1
                                          • VisualSense
                                            VisualSense Subscriber last edited by

                                            Manual is a no go for large sites. If someone knows a tool like this, it woul be cool to know which/ where to find. Or..... This would make a cool SEOmoz pro tool 🙂

                                            1 Reply Last reply Reply Quote 4
                                            • Function5
                                              Function5 @KeriMorgret last edited by

                                              My bad - you are right that it doesn't display the actual URLs. So I guess the best thing you can do is site:examplesite.com and see what comes up.

                                              1 Reply Last reply Reply Quote 0
                                              • KeriMorgret
                                                KeriMorgret @Function5 last edited by

                                                That will tell you the number indexed, but it still doesn't tell you which of those URLs are or are not indexed. I think we all wish it would!

                                                Function5 1 Reply Last reply Reply Quote 1
                                                • Function5
                                                  Function5 last edited by

                                                  I would use Google Webmaster Tools as you can see how many URLs are indexed based on your sitemap. Once you have that, you can compare it to your total list. The same can be done with Bing.

                                                  KeriMorgret 1 Reply Last reply Reply Quote 1
                                                  • elephantseo
                                                    elephantseo @DavidKauzlaric last edited by

                                                    Yeah I do it manually now so was looking for something more efficient.

                                                    1 Reply Last reply Reply Quote 0
                                                    • DavidKauzlaric
                                                      DavidKauzlaric last edited by

                                                      We built an internal tool to do it for us, but basically you can do this manually.

                                                      Go to google, type in "site:YOURURLHERE" without the quotes. You can check a certain page, a site, a subdomain, etc... of course if you have thousands of URLs this method is not ideal, but it can be done.

                                                      elephantseo Chris.McGiff 2 Replies Last reply Reply Quote 2
                                                      • 1 / 1
                                                      • First post
                                                        Last post

                                                      Got a burning SEO question?

                                                      Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                                                      Start my free trial


                                                      Browse Questions

                                                      Explore more categories

                                                      • Moz Tools

                                                        Chat with the community about the Moz tools.

                                                      • SEO Tactics

                                                        Discuss the SEO process with fellow marketers

                                                      • Community

                                                        Discuss industry events, jobs, and news!

                                                      • Digital Marketing

                                                        Chat about tactics outside of SEO

                                                      • Research & Trends

                                                        Dive into research and trends in the search industry.

                                                      • Support

                                                        Connect on product support and feature requests.

                                                      • See all categories

                                                      Related Questions

                                                      • TheHecksler

                                                        Google tries to index non existing language URLs. Why?

                                                        Hi, I am working for a SAAS client. He uses two different language versions by using two different subdomains.
                                                        de.domain.com/company for german and en.domain.com for english. Many thousands URLs has been indexed correctly. But Google Search Console tries to index URLs which were never existing before and are still not existing. de.domain.com**/en/company
                                                        en.domain.com
                                                        /de/**company ... and an thousand more using the /en/ or /de/ in between. We never use this variant and calling these URLs will throw up a 404 Page correctly (but with wrong respond code  -  we`re fixing that 😉 ). But Google tries to index these kind of URLs again and again. And, I couldnt find any source of these URLs. No Website is using this as an out going link, etc.
                                                        We do see in our logfiles, that a Screaming Frog Installation and moz.com w opensiteexplorer were trying to access this earlier. My Question: How does Google comes up with that? From where did they get these URLs, that (to our knowledge) never existed? Any ideas? Thanks 🙂

                                                        Technical SEO | | TheHecksler
                                                        0
                                                      • lfrazer123

                                                        Folders in url structure?

                                                        Hello, Revamping an out-of-date website and am wondering if I need to include the folders (categories) in the url structure? The proposed structure has 8 main folders. I've been reading that Google is ok if the folder is not included in the url, but is it really? The hesitation I have is that the urls are getting long and the main folder only has only a sub folder beneath it. So, /folder-name/facility-name/treatment-overview. This looks too long, doesn't it? Thanks!

                                                        Technical SEO | | lfrazer123
                                                        0
                                                      • zasite

                                                        Do URLs with canonical tags get indexed by Google?

                                                        Hi, we re-branded and launched a new website in February 2016. In June we saw a steep drop in the number of URLs indexed, and there have continued to be smaller dips since. We started an account with Moz and found several thousand high priority crawl errors for duplicate pages and have since fixed those with canonical tags. However, we are still seeing the number of URLs indexed drop. Do URLs with canonical tags get indexed by Google? I can't seem to find a definitive answer on this. A good portion of our URLs have canonical tags because they are just events with different dates, but otherwise the content of the page is the same.

                                                        Technical SEO | | zasite
                                                        0
                                                      • Nanook1

                                                        Is it better to use XXX.com or XXX.com/index.html as canonical page

                                                        Is it better to use 301 redirects or canonical page? I suspect canonical is easier. The question is, which is the best canonical page, YYY.com or YYY.com/indexhtml? I assume YYY.com, since there will be many other pages such as YYY.com/info.html, YYY.com/services.html, etc.

                                                        Technical SEO | | Nanook1
                                                        0
                                                      • FranFerrara

                                                        Why do some URLs for a specific client have "/index.shtml"?

                                                        Reviewing our client's URLs for a 301 redirect strategy, we have noticed that many URLs have "/index.shtml." The part we don'd understand is these URLs aren't the homepage and they have multiple folders followed by "/index.shtml" Does anyone happen to know why this may be occurring? Is there any SEO value in keeping the "/index.shtml" in the URL?

                                                        Technical SEO | | FranFerrara
                                                        0
                                                      • themegroup

                                                        Effective use of hReview

                                                        Hi fellow Mozzers! I am just in the process of adding various reviews to our site (a design agency), but I wanted to use the ratings in different ways depending on the page. So for the home page and the services (branding, POS, direct mail etc) I wanted to aggregate relevant reviews (giving us an average of all reviews for the home page, an average of ratings from all brand projects and so on). Then, I wanted to put specific reviews on our portfolio pages, so the review relates specifically to that project. This is the easiest to do as the hReview generator is geared up for reviews that come from one source, but I can't find a way of aggregating the star ratings to make an average rating rich snippet. Anyone know where I can get the coding for this? Thanks in advance! Nick.

                                                        Technical SEO | | themegroup
                                                        0
                                                      • upick-162391

                                                        Trailing Slashes In Url use Canonical Url or 301 Redirect?

                                                        I was thinking of using 301 redirects for trailing slahes to no trailing slashes for my urls. EG: www.url.com/page1/ 301 redirect to www.url.com/page1 Already got a redirect for non-www to www already. Just wondering in my case would it be best to continue using htacces for the trailing slash redirect or just go with Canonical URLs?

                                                        Technical SEO | | upick-162391
                                                        0
                                                      • RikkiD22

                                                        Best free tool to check internal broken links

                                                        Question says it all I guess. What would your recommend as the best free tool to check internal broken links?

                                                        Technical SEO | | RikkiD22
                                                        5

                                                      Get started with Moz Pro!

                                                      Unlock the power of advanced SEO tools and data-driven insights.

                                                      Start my free trial
                                                      Products
                                                      • Moz Pro
                                                      • Moz Local
                                                      • Moz API
                                                      • Moz Data
                                                      • STAT
                                                      • Product Updates
                                                      Moz Solutions
                                                      • SMB Solutions
                                                      • Agency Solutions
                                                      • Enterprise Solutions
                                                      Free SEO Tools
                                                      • Domain Authority Checker
                                                      • Link Explorer
                                                      • Keyword Explorer
                                                      • Competitive Research
                                                      • Brand Authority Checker
                                                      • MozBar Extension
                                                      • MozCast
                                                      Resources
                                                      • Blog
                                                      • SEO Learning Center
                                                      • Help Hub
                                                      • Beginner's Guide to SEO
                                                      • How-to Guides
                                                      • Moz Academy
                                                      • API Docs
                                                      About Moz
                                                      • About
                                                      • Team
                                                      • Careers
                                                      • Contact
                                                      Why Moz
                                                      • Case Studies
                                                      • Testimonials
                                                      Get Involved
                                                      • Become an Affiliate
                                                      • MozCon
                                                      • Webinars
                                                      • Practical Marketer Series
                                                      • MozPod
                                                      Connect with us

                                                      Contact the Help team

                                                      Join our newsletter
                                                      Moz logo
                                                      © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                                                      • Accessibility
                                                      • Terms of Use
                                                      • Privacy

                                                      Looks like your connection to Moz was lost, please wait while we try to reconnect.