undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Technical SEO
  4. What tool do you use to check for URLs not indexed?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

What tool do you use to check for URLs not indexed?

Technical SEO
24
25
26.2k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • elephantseo
    elephantseo last edited by Mar 4, 2011, 3:31 PM

    What is your favorite tool for getting a report of URLs that are not cached/indexed in Google & Bing for an entire site? Basically I want a list of URLs not cached in Google and a seperate list for Bing.

    Thanks,

    Mark

    Shiv24k 1 Reply Last reply Apr 6, 2024, 11:58 AM Reply Quote 3
    • austinbenz8282
      austinbenz8282 last edited by Jun 23, 2024, 6:30 PM

      I've had good results using Google Search Console for checking which URLs are indexed. It's pretty straightforward and gives a clear overview of any indexing issues halloweensquishmallows.

      1 Reply Last reply Reply Quote 0
      • topic:timeago_earlier,3 months
      • Shiv24k
        Shiv24k @elephantseo last edited by Apr 6, 2024, 11:58 AM

        @elephantseo https://docs.google.com/document/d/e/2PACX-1vQprHYh4feCfnhC72ldSnbCG0OoXBgqpKHC2bvlbs7hX3XZ8axnrY-LuIDS5-BJ_0mSsUlcT0_OyUvp/pub

        1 Reply Last reply Reply Quote 0
        • topic:timeago_earlier,3 years
        • noitcudni
          noitcudni last edited by Jun 6, 2021, 3:40 AM

          I can work on building this tool if there's enough interest.

          1 Reply Last reply Reply Quote 0
          • topic:timeago_earlier,4 months
          • Adlanera
            Adlanera last edited by Feb 19, 2021, 9:06 AM Feb 19, 2021, 9:06 AM

            I generally just use Xenu's hyperlink sleuth (if you thousands of pages) to listing out all the URLs you have got and I might then manually take a look at them, however, see the guitar in demand I have not come upon an automatic device yet. If all people are aware of any, I'd like to recognize as properly.

            1 Reply Last reply Reply Quote 0
            • topic:timeago_earlier,6 years
            • CarlLarson
              CarlLarson last edited by Oct 22, 2014, 9:12 PM Oct 22, 2014, 9:12 PM

              This post from Distilled mentions that SEO for Excel plugin has a "Indexation Checker":
              https://www.distilled.net/blog/seo/awesome-examples-of-how-to-use-seotools-for-excel/

              Alas, after downloading and installing, it appears this feature was removed...

              1 Reply Last reply Reply Quote 0
              • topic:timeago_earlier,2 years
              • CMC-SD
                CMC-SD @kdaly100 last edited by Oct 5, 2012, 2:01 PM Oct 5, 2012, 2:01 PM

                Unless I'm missing something, there doesn't seem to be a way to get Google to show more than 100 results on a page. Our site has about 8,000 pages, and I don't relish the idea of manually exporting 80 SERPs.

                1 Reply Last reply Reply Quote 0
                • topic:timeago_earlier,25 days
                • rickv101
                  rickv101 last edited by Sep 10, 2012, 6:09 PM Sep 10, 2012, 6:09 PM

                  Annie Cushing from Seer Interactive made an awesome list of all the must have tools for SEO.

                  You can get it from her link which is http://bit.ly/tools-galore

                  In the list there is a tool called scrapebox which is great for this. In fact there are many uses for the software, it is also useful for sourcing potential link partners.

                  1 Reply Last reply Reply Quote 0
                  • topic:timeago_earlier,3 months
                  • Philip_Petrescu
                    Philip_Petrescu @Chris.McGiff last edited by Jun 8, 2012, 7:31 PM Jun 8, 2012, 7:31 PM

                    I would suggest using the Website Auditor from Advanced Web Ranking. It can parse 10.000 pages and it will tell you a lot more info than just if it's indexed by Google or not.

                    1 Reply Last reply Reply Quote 0
                    • topic:timeago_earlier,about a month
                    • squareplug
                      squareplug last edited by Apr 30, 2012, 2:32 AM Apr 30, 2012, 2:32 AM

                      hmm...I thought there was a way to pull those SERPs urls into Google docs using a function of some sort?

                      1 Reply Last reply Reply Quote 0
                      • DanishWadhwa
                        DanishWadhwa last edited by Apr 27, 2012, 2:45 PM Apr 27, 2012, 2:45 PM

                        I think you need not any tool for this, you can directly go to google.com and search: Site:www.YourWebsiteNem.com Site:www.YourWebsiteName.com/directory I think this will be the best option to check if your website is crwled by google or not.

                        1 Reply Last reply Reply Quote 0
                        • topic:timeago_earlier,about a month
                        • Chris.McGiff
                          Chris.McGiff @DavidKauzlaric last edited by Mar 20, 2012, 11:10 AM Mar 20, 2012, 11:10 AM

                          I do something similar but use Advanced Web Ranking, use site:www.domain.com as your phrase, run it to retrieve 1000 results and generate a Top Site Report in Excel to get the indexed list.

                          Also remember that you can do it on sub-directories (or partial URL paths) as a way to get more than 1000 pages from the site. In general I run it once with site:www.domain.com, then identify the most frequent sub-directories, and add those as additional phrases to the project and run a second time, i.e.: site:www.domain.com site:www.domain.com/dir1 site:www.domain.com/dir2 etc.

                          Still not definitive, but think it does give indication of where value is.

                          Philip_Petrescu 1 Reply Last reply Jun 8, 2012, 7:31 PM Reply Quote 1
                          • topic:timeago_earlier,2 months
                          • lonniea
                            lonniea last edited by Jan 29, 2012, 1:21 AM Jan 29, 2012, 1:21 AM

                            David Kauzlaric has in my opinion the best answer. If google hasn't indexed it and you've investigated your Google webmaster account, then there isn't anything better out there as far as I'm concerned. It's by far the simplest, quickest and easiest way to identify a serp result.

                            re: David Kauzlaric

                            We built an internal tool to do it for us, but basically you can do this manually.

                            Go to google, type in "site:YOURURLHERE" without the quotes. You can check a certain page, a site, a subdomain, etc... of course if you have thousands of URLs this method is not ideal, but it can be done.

                            Cheers!

                            1 Reply Last reply Reply Quote -1
                            • topic:timeago_earlier,about a month
                            • ezclickmedia
                              ezclickmedia @Syed1 last edited by Dec 29, 2011, 4:58 PM Dec 29, 2011, 4:58 PM

                              I concur, Xenu is an extremely valuable tool for me that I use daily. Also, once you get a list of all the URLs on your site, you can compare the two lists in excel (two lists being the Xenu page list for your site and the list of pages that have been indexed by Google).

                              1 Reply Last reply Reply Quote 0
                              • topic:timeago_earlier,about a month
                              • ShaMenz
                                ShaMenz @kdaly100 last edited by Nov 29, 2011, 2:45 AM Nov 29, 2011, 2:45 AM

                                Nice solution Kieran!

                                I use the same method, to compare URL list from Screaming Frog output with URL Found column from my Keyword Ranking tool - of course it doesn't catch all pages that might be indexed.

                                The intention is not really to get a complete list, more to "draught" out pages that need work.

                                1 Reply Last reply Reply Quote 0
                                • topic:timeago_earlier,2 months
                                • NakulGoyal
                                  NakulGoyal @kdaly100 last edited by Sep 16, 2011, 4:51 PM Sep 16, 2011, 4:51 PM

                                  I agree, this is not automated but so far, from what we know, looks like a nice and clean option. Thanks.

                                  1 Reply Last reply Reply Quote 0
                                  • kdaly100
                                    kdaly100 last edited by Sep 13, 2011, 6:03 PM Sep 13, 2011, 6:03 PM

                                    Saw this and tried the following which isn't automated but is one way of doing it.

                                    • First install SEO Quake plugin
                                    • Go to Google
                                    • Turn off Google Instant (http://www.google.com/preferences)
                                    • Go to Advanced search set the number of results you want displayed (estimate the number of pages on your site)
                                    • Then run your site:www.example.com search query
                                    • Export this to CSV
                                    • Import to Excel
                                    • Once then do a Data to columns conversion using ; as a delimiter (this is the CSV delimiter)
                                    • This gives you a formatted list.
                                    • Then import your sitemap.xml into another TAB in Excel
                                    • Run a vlookup between the URL tabs to flag which are on sitemap or vice versa.

                                    Not exactly automated but does the job.

                                    NakulGoyal ShaMenz CMC-SD 3 Replies Last reply Oct 5, 2012, 2:01 PM Reply Quote 15
                                    • topic:timeago_earlier,21 days
                                    • Hakkasan
                                      Hakkasan last edited by Aug 23, 2011, 7:49 PM Aug 23, 2011, 7:49 PM

                                      Curious about this question also, it would be very useful to see a master list of all URLs on our site that are not indexed by Google so that we can take action to see what aspects of the page are lacking and what we need for it to get indexed.

                                      1 Reply Last reply Reply Quote 0
                                      • topic:timeago_earlier,19 days
                                      • Syed1
                                        Syed1 last edited by Aug 4, 2011, 4:51 PM Aug 4, 2011, 4:49 PM

                                        I usually just use Xenu's link sleuth (if you thousands of pages) to list out all the URLs you have and I would then manually check them, but I haven't come across an automated tool yet. If anyone knows any, I'd love to know as well.

                                        ezclickmedia 1 Reply Last reply Dec 29, 2011, 4:58 PM Reply Quote 1
                                        • topic:timeago_earlier,about a month
                                        • VisualSense
                                          VisualSense Subscriber last edited by Jun 22, 2011, 7:11 PM Jun 22, 2011, 7:11 PM

                                          Manual is a no go for large sites. If someone knows a tool like this, it woul be cool to know which/ where to find. Or..... This would make a cool SEOmoz pro tool 🙂

                                          1 Reply Last reply Reply Quote 4
                                          • Function5
                                            Function5 @KeriMorgret last edited by Jun 20, 2011, 10:52 PM Jun 20, 2011, 10:52 PM

                                            My bad - you are right that it doesn't display the actual URLs. So I guess the best thing you can do is site:examplesite.com and see what comes up.

                                            1 Reply Last reply Reply Quote 0
                                            • KeriMorgret
                                              KeriMorgret @Function5 last edited by Jun 20, 2011, 10:38 PM Jun 20, 2011, 10:38 PM

                                              That will tell you the number indexed, but it still doesn't tell you which of those URLs are or are not indexed. I think we all wish it would!

                                              Function5 1 Reply Last reply Jun 20, 2011, 10:52 PM Reply Quote 1
                                              • Function5
                                                Function5 last edited by Jun 20, 2011, 10:21 PM Jun 20, 2011, 10:21 PM

                                                I would use Google Webmaster Tools as you can see how many URLs are indexed based on your sitemap. Once you have that, you can compare it to your total list. The same can be done with Bing.

                                                KeriMorgret 1 Reply Last reply Jun 20, 2011, 10:38 PM Reply Quote 1
                                                • topic:timeago_earlier,4 months
                                                • elephantseo
                                                  elephantseo @DavidKauzlaric last edited by Mar 4, 2011, 5:04 PM Mar 4, 2011, 5:04 PM

                                                  Yeah I do it manually now so was looking for something more efficient.

                                                  1 Reply Last reply Reply Quote 0
                                                  • DavidKauzlaric
                                                    DavidKauzlaric last edited by Mar 4, 2011, 4:42 PM Mar 4, 2011, 4:42 PM

                                                    We built an internal tool to do it for us, but basically you can do this manually.

                                                    Go to google, type in "site:YOURURLHERE" without the quotes. You can check a certain page, a site, a subdomain, etc... of course if you have thousands of URLs this method is not ideal, but it can be done.

                                                    elephantseo Chris.McGiff 2 Replies Last reply Mar 4, 2011, 5:04 PM Reply Quote 2
                                                    • 1 / 1
                                                    1 out of 25
                                                    • First post
                                                      1/25
                                                      Last post

                                                    Got a burning SEO question?

                                                    Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                                                    Start my free trial


                                                    Browse Questions

                                                    Explore more categories

                                                    • Moz Tools

                                                      Chat with the community about the Moz tools.

                                                    • SEO Tactics

                                                      Discuss the SEO process with fellow marketers

                                                    • Community

                                                      Discuss industry events, jobs, and news!

                                                    • Digital Marketing

                                                      Chat about tactics outside of SEO

                                                    • Research & Trends

                                                      Dive into research and trends in the search industry.

                                                    • Support

                                                      Connect on product support and feature requests.

                                                    • See all categories

                                                    Related Questions

                                                    • ghesta

                                                      In writing the url, it is better to use the language used by the people of my country or English?

                                                      We speak Persian and all people search in Persian on Google. But I read in some sources that the url should be in English. Please tell me which language to use for url writing?
                                                      For example, I brought down two models: 1fb0e134-10dc-4737-904f-bfdf07143a98-image.png https://ghesta.ir/blog/how-to-become-rich/
                                                      2)https://ghesta.ir/blog/چگونه-پولدار-شویم/

                                                      Technical SEO | Jun 10, 2021, 10:32 PM | ghesta
                                                      0
                                                    • tiromedia

                                                      Should we use Cloudflare

                                                      Hi all, we want to speed up our website (hosted in Wordpress, traffic around 450,000 page views monthly), we use lots of images. And we're wondering about setting up on Cloudflare, however after searching a bit in Google I have seen some people say the change in IP, or possible sharing of Its with bad neighbourhoods, can really hit search rankings. So, I was wondering what the latest thinking is on this subject, would the increased speed and local server locations be a boost for SEO, moreso than a potential loss of rankings for changing IP? Thanks!

                                                      Technical SEO | Jun 3, 2020, 2:20 AM | tiromedia
                                                      1
                                                    • muzzmoz

                                                      Indexed pages

                                                      Just started a site audit and trying to determine the number of pages on a client site and whether there are more pages being indexed than actually exist. I've used four tools and got four very different answers... Google Search Console: 237 indexed pages Google search using site command: 468 results MOZ site crawl: 1013 unique URLs Screaming Frog: 183 page titles, 187 URIs (note this is a free licence, but should cut off at 500) Can anyone shed any light on why they differ so much? And where lies the truth?

                                                      Technical SEO | Oct 30, 2016, 3:12 PM | muzzmoz
                                                      1
                                                    • MichaelGregory

                                                      Bulk URL Removal in Webmaster Tools

                                                      One of Wordpress sites was hacked (for about 10 hours), and Google picked up 4000+ urls in the index. The site is fixed, but I'm stuck with all those urls in the index. All the urls of of the form: walkerorthodontics.com/index.php?online-payday-cash-loan.htmloncewe The only bulk removal option I could find was to remove an entire folder, but I can't do that, as it would only leave the homepage and kill off everything else. For some crazy reason, the removal tools doesn't support wildcards, so that obvious solution is right out. So, how do it get rid of 4000 results? And no, waiting around for them to 404 out of the index isn't an option.

                                                      Technical SEO | Jun 9, 2016, 6:29 PM | MichaelGregory
                                                      0
                                                    • Deacyde

                                                      Canonical homepage link uses trailing slash while default homepage uses no trailing slash, will this be an issue?

                                                      Hello, 1st off, let me explain my client in this case uses BigCommerce, and I don't have access to the backend like most other situations. So I have to rely on BG to handle certain issues. I'm curious if there is much of a difference using domain.com/ as the canonical url while BG currently is redirecting our domain to domain.com. I've been using domain.com/ consistently for the last 6 months, and since we switches stores on Friday, this issue has popped up and has me a bit worried that we'll loose somehow via link juice or overall indexing since this could confuse crawlers. Now some say that the domain url is fine using / or not, as per - https://moz.com/community/q/trailing-slash-and-rel-canonical But I also wanted to see what you all felt about this. What says you?

                                                      Technical SEO | Mar 8, 2016, 8:35 PM | Deacyde
                                                      0
                                                    • linklander

                                                      How to check if an individual page is indexed by Google?

                                                      So my understanding is that you can use site: [page url without http] to check if a page is indexed by Google, is this 100% reliable though? Just recently Ive worked on a few pages that have not shown up when Ive checked them using site: but they do show up when using info: and also show their cached versions, also the rest of the site and pages above it (the url I was checking was quite deep) are indexed just fine. What does this mean? thank you p.s I do not have WMT or GA access for these sites

                                                      Technical SEO | Aug 25, 2015, 6:45 PM | linklander
                                                      0
                                                    • Nanook1

                                                      Is it better to use XXX.com or XXX.com/index.html as canonical page

                                                      Is it better to use 301 redirects or canonical page? I suspect canonical is easier. The question is, which is the best canonical page, YYY.com or YYY.com/indexhtml? I assume YYY.com, since there will be many other pages such as YYY.com/info.html, YYY.com/services.html, etc.

                                                      Technical SEO | Jan 2, 2015, 7:27 PM | Nanook1
                                                      0
                                                    • fabiank

                                                      Index forum sites

                                                      Hi Moz Team, somehow the last question i raised a few days ago not only wasnt answered up until now, it was also completely deleted and the credit was not "refunded" - obviously there was some data loss involved with your restructuring. Can you check whether you still find the last question and answer it quickly? I need the answer 🙂 Here is one more question: I bought a website that has a huge forum, loads of pages with user generated content. Overall around 500.000 Threads with 9 Million comments. The complete forum is noindex/nofollow when i bought the site, now i am thinking about what is the best way to unleash the potential. The current system is vBulletin 3.6.10. a) Shall i first do an update of vbulletin to version 4 and use the vSEO tool to make the URLs clean, more user and search engine friendly before i switch to index/follow? b) would you recommend to have the forum in the folder structure or on a subdomain? As far as i know subdomain does take lesser strenght from the TLD, however, it is safer because the subdomain is seen as a separate entity from the regular TLD. Having it in he folder makes it easiert to pass strenght from the TLD to the forum, however, it puts my TLD at risk c) Would you release all forum sites at once or section by section? I think section by section looks rather unnatural not only to search engines but also to users, however, i am afraid of blasting more than a millionpages into the index at once. d) Would you index the first page of a threat or all pages of a threat? I fear duplicate content as the different pages of the threat contain different body content but the same Title and possibly the same h1. Looking forward to hear from you soon! Best Fabian

                                                      Technical SEO | May 17, 2011, 6:51 PM | fabiank
                                                      0

                                                    Get started with Moz Pro!

                                                    Unlock the power of advanced SEO tools and data-driven insights.

                                                    Start my free trial
                                                    Products
                                                    • Moz Pro
                                                    • Moz Local
                                                    • Moz API
                                                    • Moz Data
                                                    • STAT
                                                    • Product Updates
                                                    Moz Solutions
                                                    • SMB Solutions
                                                    • Agency Solutions
                                                    • Enterprise Solutions
                                                    Free SEO Tools
                                                    • Domain Authority Checker
                                                    • Link Explorer
                                                    • Keyword Explorer
                                                    • Competitive Research
                                                    • Brand Authority Checker
                                                    • Local Citation Checker
                                                    • MozBar Extension
                                                    • MozCast
                                                    Resources
                                                    • Blog
                                                    • SEO Learning Center
                                                    • Help Hub
                                                    • Beginner's Guide to SEO
                                                    • How-to Guides
                                                    • Moz Academy
                                                    • API Docs
                                                    About Moz
                                                    • About
                                                    • Team
                                                    • Careers
                                                    • Contact
                                                    Why Moz
                                                    • Case Studies
                                                    • Testimonials
                                                    Get Involved
                                                    • Become an Affiliate
                                                    • MozCon
                                                    • Webinars
                                                    • Practical Marketer Series
                                                    • MozPod
                                                    Connect with us

                                                    Contact the Help team

                                                    Join our newsletter
                                                    Moz logo
                                                    © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                                                    • Accessibility
                                                    • Terms of Use
                                                    • Privacy

                                                    Looks like your connection to Moz was lost, please wait while we try to reconnect.