Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • SEO Q&A
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • Case Studies
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      What is your Brand Authority?
      Moz

      What is your Brand Authority?

      Check yours now
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • SEO Q&A

        Insights & discussions from an SEO community of 500,000+.

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • Case Studies

        Explore how Moz drives ROI with a proven track record of success.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Technical SEO
    4. What tool do you use to check for URLs not indexed?

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    What tool do you use to check for URLs not indexed?

    Technical SEO
    24
    25
    26190
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • elephantseo
      elephantseo last edited by

      What is your favorite tool for getting a report of URLs that are not cached/indexed in Google & Bing for an entire site? Basically I want a list of URLs not cached in Google and a seperate list for Bing.

      Thanks,

      Mark

      Shiv24k 1 Reply Last reply Reply Quote 3
      • austinbenz8282
        austinbenz8282 last edited by

        I've had good results using Google Search Console for checking which URLs are indexed. It's pretty straightforward and gives a clear overview of any indexing issues halloweensquishmallows.

        1 Reply Last reply Reply Quote 0
        • Shiv24k
          Shiv24k @elephantseo last edited by

          @elephantseo https://docs.google.com/document/d/e/2PACX-1vQprHYh4feCfnhC72ldSnbCG0OoXBgqpKHC2bvlbs7hX3XZ8axnrY-LuIDS5-BJ_0mSsUlcT0_OyUvp/pub

          1 Reply Last reply Reply Quote 0
          • noitcudni
            noitcudni last edited by

            I can work on building this tool if there's enough interest.

            1 Reply Last reply Reply Quote 0
            • Adlanera
              Adlanera last edited by

              I generally just use Xenu's hyperlink sleuth (if you thousands of pages) to listing out all the URLs you have got and I might then manually take a look at them, however, see the guitar in demand I have not come upon an automatic device yet. If all people are aware of any, I'd like to recognize as properly.

              1 Reply Last reply Reply Quote 0
              • CarlLarson
                CarlLarson last edited by

                This post from Distilled mentions that SEO for Excel plugin has a "Indexation Checker":
                https://www.distilled.net/blog/seo/awesome-examples-of-how-to-use-seotools-for-excel/

                Alas, after downloading and installing, it appears this feature was removed...

                1 Reply Last reply Reply Quote 0
                • CMC-SD
                  CMC-SD @kdaly100 last edited by

                  Unless I'm missing something, there doesn't seem to be a way to get Google to show more than 100 results on a page. Our site has about 8,000 pages, and I don't relish the idea of manually exporting 80 SERPs.

                  1 Reply Last reply Reply Quote 0
                  • rickv101
                    rickv101 last edited by

                    Annie Cushing from Seer Interactive made an awesome list of all the must have tools for SEO.

                    You can get it from her link which is http://bit.ly/tools-galore

                    In the list there is a tool called scrapebox which is great for this. In fact there are many uses for the software, it is also useful for sourcing potential link partners.

                    1 Reply Last reply Reply Quote 0
                    • Philip_Petrescu
                      Philip_Petrescu @Chris.McGiff last edited by

                      I would suggest using the Website Auditor from Advanced Web Ranking. It can parse 10.000 pages and it will tell you a lot more info than just if it's indexed by Google or not.

                      1 Reply Last reply Reply Quote 0
                      • squareplug
                        squareplug last edited by

                        hmm...I thought there was a way to pull those SERPs urls into Google docs using a function of some sort?

                        1 Reply Last reply Reply Quote 0
                        • DanishWadhwa
                          DanishWadhwa last edited by

                          I think you need not any tool for this, you can directly go to google.com and search: Site:www.YourWebsiteNem.com Site:www.YourWebsiteName.com/directory I think this will be the best option to check if your website is crwled by google or not.

                          1 Reply Last reply Reply Quote 0
                          • Chris.McGiff
                            Chris.McGiff @DavidKauzlaric last edited by

                            I do something similar but use Advanced Web Ranking, use site:www.domain.com as your phrase, run it to retrieve 1000 results and generate a Top Site Report in Excel to get the indexed list.

                            Also remember that you can do it on sub-directories (or partial URL paths) as a way to get more than 1000 pages from the site. In general I run it once with site:www.domain.com, then identify the most frequent sub-directories, and add those as additional phrases to the project and run a second time, i.e.: site:www.domain.com site:www.domain.com/dir1 site:www.domain.com/dir2 etc.

                            Still not definitive, but think it does give indication of where value is.

                            Philip_Petrescu 1 Reply Last reply Reply Quote 1
                            • lonniea
                              lonniea last edited by

                              David Kauzlaric has in my opinion the best answer. If google hasn't indexed it and you've investigated your Google webmaster account, then there isn't anything better out there as far as I'm concerned. It's by far the simplest, quickest and easiest way to identify a serp result.

                              re: David Kauzlaric

                              We built an internal tool to do it for us, but basically you can do this manually.

                              Go to google, type in "site:YOURURLHERE" without the quotes. You can check a certain page, a site, a subdomain, etc... of course if you have thousands of URLs this method is not ideal, but it can be done.

                              Cheers!

                              1 Reply Last reply Reply Quote -1
                              • ezclickmedia
                                ezclickmedia @Syed1 last edited by

                                I concur, Xenu is an extremely valuable tool for me that I use daily. Also, once you get a list of all the URLs on your site, you can compare the two lists in excel (two lists being the Xenu page list for your site and the list of pages that have been indexed by Google).

                                1 Reply Last reply Reply Quote 0
                                • ShaMenz
                                  ShaMenz @kdaly100 last edited by

                                  Nice solution Kieran!

                                  I use the same method, to compare URL list from Screaming Frog output with URL Found column from my Keyword Ranking tool - of course it doesn't catch all pages that might be indexed.

                                  The intention is not really to get a complete list, more to "draught" out pages that need work.

                                  1 Reply Last reply Reply Quote 0
                                  • NakulGoyal
                                    NakulGoyal @kdaly100 last edited by

                                    I agree, this is not automated but so far, from what we know, looks like a nice and clean option. Thanks.

                                    1 Reply Last reply Reply Quote 0
                                    • kdaly100
                                      kdaly100 last edited by

                                      Saw this and tried the following which isn't automated but is one way of doing it.

                                      • First install SEO Quake plugin
                                      • Go to Google
                                      • Turn off Google Instant (http://www.google.com/preferences)
                                      • Go to Advanced search set the number of results you want displayed (estimate the number of pages on your site)
                                      • Then run your site:www.example.com search query
                                      • Export this to CSV
                                      • Import to Excel
                                      • Once then do a Data to columns conversion using ; as a delimiter (this is the CSV delimiter)
                                      • This gives you a formatted list.
                                      • Then import your sitemap.xml into another TAB in Excel
                                      • Run a vlookup between the URL tabs to flag which are on sitemap or vice versa.

                                      Not exactly automated but does the job.

                                      NakulGoyal ShaMenz CMC-SD 3 Replies Last reply Reply Quote 15
                                      • Hakkasan
                                        Hakkasan last edited by

                                        Curious about this question also, it would be very useful to see a master list of all URLs on our site that are not indexed by Google so that we can take action to see what aspects of the page are lacking and what we need for it to get indexed.

                                        1 Reply Last reply Reply Quote 0
                                        • Syed1
                                          Syed1 last edited by

                                          I usually just use Xenu's link sleuth (if you thousands of pages) to list out all the URLs you have and I would then manually check them, but I haven't come across an automated tool yet. If anyone knows any, I'd love to know as well.

                                          ezclickmedia 1 Reply Last reply Reply Quote 1
                                          • VisualSense
                                            VisualSense Subscriber last edited by

                                            Manual is a no go for large sites. If someone knows a tool like this, it woul be cool to know which/ where to find. Or..... This would make a cool SEOmoz pro tool 🙂

                                            1 Reply Last reply Reply Quote 4
                                            • Function5
                                              Function5 @KeriMorgret last edited by

                                              My bad - you are right that it doesn't display the actual URLs. So I guess the best thing you can do is site:examplesite.com and see what comes up.

                                              1 Reply Last reply Reply Quote 0
                                              • KeriMorgret
                                                KeriMorgret @Function5 last edited by

                                                That will tell you the number indexed, but it still doesn't tell you which of those URLs are or are not indexed. I think we all wish it would!

                                                Function5 1 Reply Last reply Reply Quote 1
                                                • Function5
                                                  Function5 last edited by

                                                  I would use Google Webmaster Tools as you can see how many URLs are indexed based on your sitemap. Once you have that, you can compare it to your total list. The same can be done with Bing.

                                                  KeriMorgret 1 Reply Last reply Reply Quote 1
                                                  • elephantseo
                                                    elephantseo @DavidKauzlaric last edited by

                                                    Yeah I do it manually now so was looking for something more efficient.

                                                    1 Reply Last reply Reply Quote 0
                                                    • DavidKauzlaric
                                                      DavidKauzlaric last edited by

                                                      We built an internal tool to do it for us, but basically you can do this manually.

                                                      Go to google, type in "site:YOURURLHERE" without the quotes. You can check a certain page, a site, a subdomain, etc... of course if you have thousands of URLs this method is not ideal, but it can be done.

                                                      elephantseo Chris.McGiff 2 Replies Last reply Reply Quote 2
                                                      • 1 / 1
                                                      • First post
                                                        Last post

                                                      Got a burning SEO question?

                                                      Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                                                      Start my free trial


                                                      Browse Questions

                                                      Explore more categories

                                                      • Moz Tools

                                                        Chat with the community about the Moz tools.

                                                      • SEO Tactics

                                                        Discuss the SEO process with fellow marketers

                                                      • Community

                                                        Discuss industry events, jobs, and news!

                                                      • Digital Marketing

                                                        Chat about tactics outside of SEO

                                                      • Research & Trends

                                                        Dive into research and trends in the search industry.

                                                      • Support

                                                        Connect on product support and feature requests.

                                                      • See all categories

                                                      Related Questions

                                                      • Hermski

                                                        Japanese URL-structured sitemap (pages) not being indexed by Bing Webmaster Tools

                                                        Hello everyone, I am facing an issue with the sitemap submission feature in Bing Webmaster Tools for a Japanese language subdirectory domain project. Just to outline the key points: The website is based on a subdirectory URL ( example.com/ja/ ) The Japanese URLs (when pages are published in WordPress) are not being encoded. They are entered in pure Kanji. Google Webmaster Tools, for instance, has no issues reading and indexing the page's URLs in its sitemap submission area (all pages are being indexed). When it comes to Bing Webmaster Tools it's a different story, though. Basically, after the sitemap has been submitted ( example.com/ja/sitemap.xml ), it does report an error that it failed to download this part of the sitemap: "page-sitemap.xml" (basically the sitemap featuring all the sites pages). That means that no URLs have been submitted to Bing either. My apprehension is that Bing Webmaster Tools does not understand the Japanese URLs (or the Kanji for that matter). Therefore, I generally wonder what the correct way is to go on about this. When viewing the sitemap ( example.com/ja/page-sitemap.xml ) in a web browser, though, the Japanese URL's characters are already displayed as encoded. I am not sure if submitting the Kanji style URLs separately is a solution. In Bing Webmaster Tools this can only be done on the root domain level ( example.com ). However, surely there must be a way to make Bing's sitemap submission understand Japanese style sitemaps? Many thanks everyone for any advice!

                                                        Technical SEO | | Hermski
                                                        0
                                                      • TheHecksler

                                                        Google tries to index non existing language URLs. Why?

                                                        Hi, I am working for a SAAS client. He uses two different language versions by using two different subdomains.
                                                        de.domain.com/company for german and en.domain.com for english. Many thousands URLs has been indexed correctly. But Google Search Console tries to index URLs which were never existing before and are still not existing. de.domain.com**/en/company
                                                        en.domain.com
                                                        /de/**company ... and an thousand more using the /en/ or /de/ in between. We never use this variant and calling these URLs will throw up a 404 Page correctly (but with wrong respond code  -  we`re fixing that 😉 ). But Google tries to index these kind of URLs again and again. And, I couldnt find any source of these URLs. No Website is using this as an out going link, etc.
                                                        We do see in our logfiles, that a Screaming Frog Installation and moz.com w opensiteexplorer were trying to access this earlier. My Question: How does Google comes up with that? From where did they get these URLs, that (to our knowledge) never existed? Any ideas? Thanks 🙂

                                                        Technical SEO | | TheHecksler
                                                        0
                                                      • Tom3_15

                                                        Google is indexing bad URLS

                                                        Hi All, The site I am working on is built on Wordpress. The plugin Revolution Slider was downloaded. While no longer utilized, it still remained on the site for some time. This plugin began creating hundreds of URLs containing nothing but code on the page. I noticed these URLs were being indexed by Google. The URLs follow the structure: www.mysite.com/wp-content/uploads/revslider/templates/this-part-changes/ I have done the following to prevent these URLs from being created & indexed: 1. Added a directive in my Htaccess to 404 all of these URLs 2. Blocked /wp-content/uploads/revslider/ in my robots.txt 3. Manually de-inedex each URL using the GSC tool 4. Deleted the plugin However, new URLs still appear in Google's index, despite being blocked by robots.txt and resolving to a 404. Can anyone suggest any next steps? I Thanks!

                                                        Technical SEO | | Tom3_15
                                                        0
                                                      • Tom3_15

                                                        Sudden Indexation of "Index of /wp-content/uploads/"

                                                        Hi all, I have suddenly noticed a massive jump in indexed pages. After performing a "site:" search, it was revealed that the sudden jump was due to the indexation of many pages beginning with the serp title "Index of /wp-content/uploads/" for many uploaded pieces of content & plugins. This has appeared approximately one month after switching to https. I have also noticed a decline in Bing rankings. Does anyone know what is causing/how to fix this? To be clear, these pages are **not **normal /wp-content/uploads/ but rather "index of" pages, being included in Google. Thank you.

                                                        Technical SEO | | Tom3_15
                                                        0
                                                      • jayoliverwright

                                                        Tools/Software that can crawl all image URLs in a site

                                                        Excluding Screaming Frog, what other tools/software to use in order to crawl all image URLs in a site? Because in Screaming Frog, they don't crawl image URLs which are not under the site domain. Example of an image URL outside the client site: http://cdn.shopify.com/images/this-is-just-a-sample.png If the client is: http://www.example.com, Screaming Frog only crawls images under it like, http://www.example.com/images/this-is-just-a-sample.png

                                                        Technical SEO | | jayoliverwright
                                                        0
                                                      • MonicaOConnor

                                                        No Index PDFs

                                                        Our products have about 4 PDFs a piece, which really inflates our indexed pages. I was wondering if I could add REL=No Index to the PDF's URL? All of the files are on a file server, so they are embedded with links on our product pages. I know I could add a No Follow attribute, but I was wondering if any one knew if the No Index would work the same or if that is even possible. Thanks!

                                                        Technical SEO | | MonicaOConnor
                                                        0
                                                      • Creditsafe

                                                        No index on subdomains

                                                        Hi, We have a subdomain that is appearing in the search results - I want to hide this as it looks really bad. If I were to add the no index tag to the sub domain would URL would this affect the whole domain or just that sub domain? The main domain is vitally important - it is just that sub domain I need to hide. Many thanks

                                                        Technical SEO | | Creditsafe
                                                        0
                                                      • themegroup

                                                        Effective use of hReview

                                                        Hi fellow Mozzers! I am just in the process of adding various reviews to our site (a design agency), but I wanted to use the ratings in different ways depending on the page. So for the home page and the services (branding, POS, direct mail etc) I wanted to aggregate relevant reviews (giving us an average of all reviews for the home page, an average of ratings from all brand projects and so on). Then, I wanted to put specific reviews on our portfolio pages, so the review relates specifically to that project. This is the easiest to do as the hReview generator is geared up for reviews that come from one source, but I can't find a way of aggregating the star ratings to make an average rating rich snippet. Anyone know where I can get the coding for this? Thanks in advance! Nick.

                                                        Technical SEO | | themegroup
                                                        0

                                                      Get started with Moz Pro!

                                                      Unlock the power of advanced SEO tools and data-driven insights.

                                                      Start my free trial
                                                      Products
                                                      • Moz Pro
                                                      • Moz Local
                                                      • Moz API
                                                      • Moz Data
                                                      • STAT
                                                      • Product Updates
                                                      Moz Solutions
                                                      • SMB Solutions
                                                      • Agency Solutions
                                                      • Enterprise Solutions
                                                      Free SEO Tools
                                                      • Domain Authority Checker
                                                      • Link Explorer
                                                      • Keyword Explorer
                                                      • Competitive Research
                                                      • Brand Authority Checker
                                                      • MozBar Extension
                                                      • MozCast
                                                      Resources
                                                      • Blog
                                                      • SEO Learning Center
                                                      • Help Hub
                                                      • Beginner's Guide to SEO
                                                      • How-to Guides
                                                      • Moz Academy
                                                      • API Docs
                                                      About Moz
                                                      • About
                                                      • Team
                                                      • Careers
                                                      • Contact
                                                      Why Moz
                                                      • Case Studies
                                                      • Testimonials
                                                      Get Involved
                                                      • Become an Affiliate
                                                      • MozCon
                                                      • Webinars
                                                      • Practical Marketer Series
                                                      • MozPod
                                                      Connect with us

                                                      Contact the Help team

                                                      Join our newsletter
                                                      Moz logo
                                                      © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                                                      • Accessibility
                                                      • Terms of Use
                                                      • Privacy

                                                      Looks like your connection to Moz was lost, please wait while we try to reconnect.