undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • MozCon
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Digital Marketers
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    Let your business shine with Listings AI
    Moz Local

    Let your business shine with Listings AI

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Digital Marketers

      Simplify SEO tasks to save time and grow your traffic.

    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Technical SEO
  4. Include or exclude noindex urls in sitemap?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Include or exclude noindex urls in sitemap?

Technical SEO
4
11
22.8k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • vcj
    vcj last edited by May 24, 2016, 5:32 PM

    We just added tags to our pages with thin content.

    Should we include or exclude those urls from our sitemap.xml file? I've read conflicting recommendations.

    1 Reply Last reply Reply Quote 0
    • SilverStar1
      SilverStar1 last edited by Oct 20, 2016, 4:30 AM Oct 20, 2016, 4:30 AM

      Hi vcj and the rest of you guys

      I would be very interested in learning what strategy you actually went ahead with, and the results.   I have a similar issue as a result of pruning, and removing noindex pages from the sitemap makes perfect sense to me.  We set a noindexed follow on several thousand pages without product descriptions/thin content and we have set things up so when we add new descriptions and updated onpage elements, the noindex is automatically reversed; which sounds perfect, however hardly any of the pages to date (3000-4000) are indexed, so looking for a feasible solution for exactly the same reasons as you.

      We have better and comparable metrics and optimization than a lot of the competition, yet rankings are mediocre, so looking to improve on this.

      It would be good to hear your views

      Cheers

      1 Reply Last reply Reply Quote 0
      • topic:timeago_earlier,5 months
      • vcj
        vcj @Yoav-Blustein last edited by May 26, 2016, 2:12 PM May 26, 2016, 2:12 PM

        I'm aware of the fact Google will get to them sooner or later.

        The recommendation from Gary Illyes (from Google), as mentioned in this post, was the reason for my asking the question. Not trying to outsmart Google, just trying to work within their guidelines in the most efficient way possible.

        1 Reply Last reply Reply Quote 0
        • Yoav-Blustein
          Yoav-Blustein last edited by May 26, 2016, 4:03 AM May 26, 2016, 4:03 AM

          Just to put things into perspective,

          if these URLs are all already indexed and you have used "noindex" on those pages, sooner or later google will re-crawl these pages and they will be removed. You may want to remove them from the index ASAP for some reason, but it wont really change anything. Because Google will not deindex your noindex pages just because they are in your sitemap.xml.
          Google deindexes a sie only when it is time to re-crawl the page.

          Google never recommends using noindex in sitemaps, and google wont suggest that in their blocking search indexing results guidelines. Also Google indicates the following:
          "Google will completely drop the page from search results, even if other pages link to it. If the content is currently in our index, we will remove it after the next time we crawl it. (To expedite removal, use the Remove URLs tool in Google Webmaster Tools.)"

          But hey! every SEO has its own take.. Some tend to try outsmart Google some not..

          Good luck

          vcj 1 Reply Last reply May 26, 2016, 2:12 PM Reply Quote 1
          • MikeRoberts
            MikeRoberts @vcj last edited by May 25, 2016, 5:07 PM May 25, 2016, 4:53 PM

            That opens up other potential restrictions to getting this done quickly and easily. I wouldn't consider it best practices to create what is essentially a spam page full of internal links and Googlebot will likely not crawl all 4000 links if you have them all there. So now you'd be talking about maybe making 20 or so thin, spammy looking pages of 200+ internal links to hopefully fix the issue.

            The quick, easy sounding options are not often the best option. Considering you're doing all of this in an attempt to fix issues that arose due to an algorithmic penalty, I'd suggest trying to follow best practices for making these changes. It might not be easy but it'll lessen your chances of having done a quick fix that might be the cause, or part of, a future penalty.

            So if Fetch As won't work for you (considering lack of manpower to manually fetch 4000 pages), the sitemap.xml option might be the better choice for you.

            1 Reply Last reply Reply Quote 1
            • vcj
              vcj @MikeRoberts last edited by May 25, 2016, 4:08 PM May 25, 2016, 4:08 PM

              Thanks, Mike.

              What are your thoughts on creating a page with links to all of the pages we've Noindexed, doing a Fetch As and submitting that URL and its linked pages? Do you think Google would dislike that?

              MikeRoberts 1 Reply Last reply May 25, 2016, 4:53 PM Reply Quote 0
              • MikeRoberts
                MikeRoberts @vcj last edited by May 25, 2016, 5:07 PM May 25, 2016, 4:01 PM

                You could technically add them to the sitemap.xml in the hopes that this will get them noticed faster but the sitemap is commonly used for the things you want Google to crawl and index. Plus, placing them in the sitemap does not guarantee Google is going to get around to crawling your change or those specific pages. Technically speaking, doing nothing and jut waiting is equally as valid. Google will recrawl your site at some point. Sitemap.xml only helps if Google is crawling you to see it. Fetch As makes Google see your page as it is now which is like forcing part of a crawl. So technically Fetch As will be the more reliable, quicker choice though it will be more labor-intensive. If you don't have the man-hours to do a project like that at the moment, then waiting or using the Sitemap could work for you. Google even suggests using Fetch As for urls you want them to see that you have blocked with meta tags: https://support.google.com/webmasters/answer/93710?hl=en&ref_topic=4598466

                vcj 1 Reply Last reply May 25, 2016, 4:08 PM Reply Quote 1
                • vcj
                  vcj @MikeRoberts last edited by May 25, 2016, 3:11 PM May 25, 2016, 3:08 PM

                  There are too many pages to do that (unless we created a page with links to all of the Noindexed pages, then asked Google to crawl that and all linked pages, though that seems like it might be a bad approach). It's an ecommerce website and we Noindexed nearly 4,000 pages that had thin or duplicate content (manufacturer descriptions, no description on brand page, etc) and had no organic traffic in the past 90 days.

                  This site was hit by Panda in September 2014 and isn't ranking for things it should be – pages with better backlink profiles, higher DA/PA, better content, etc. than our competitors. Our thought is we're not ranking because of a penalty against thin/duplicate content. So we decided to Noindex these pages, improve the content on products that are selling and getting traffic, then work on improving pages that we've Noindex before switching them back to Index.

                  Basically following recommendations from this article: https://moz.com/blog/pruning-your-ecommerce-site

                  MikeRoberts 1 Reply Last reply May 25, 2016, 4:01 PM Reply Quote 0
                  • MikeRoberts
                    MikeRoberts last edited by May 25, 2016, 2:17 PM May 25, 2016, 2:17 PM

                    If the pages are in the index and you've recently added a NoIndex tag with the express purpose of getting them removed from the index, you may be better served doing crawl requests in Search Console of the pages in question.

                    vcj 1 Reply Last reply May 25, 2016, 3:08 PM Reply Quote 0
                    • vcj
                      vcj last edited by May 25, 2016, 1:54 PM May 25, 2016, 1:54 PM

                      Thanks for your response!

                      I did some more digging. This seems to contradict your suggestion:

                      https://twitter.com/methode/status/653980524264878080

                      If the goal is to have these pages removed from the index, and having them in the sitemap means they'll be picked up sooner by Google's crawler, then it seems to make sense that they should be included until they're removed from the index.

                      Am I misinterpreting this?

                      1 Reply Last reply Reply Quote 0
                      • Yoav-Blustein
                        Yoav-Blustein last edited by May 25, 2016, 3:51 AM May 25, 2016, 3:49 AM

                        Hi

                        The reason you submit a sitemap to a searchengine is to ease and aid in crawling process for the pages that you want to get indexed. It speeds up the crawling process and lets search engine to discover all those pages that has no inner linkings to it  etc..

                        A "noindex" tag does the opposite.
                        So no, you should not include noindex pages inside your sitemap files. 
                        In general you should avoid pages that are not returning 200 also.

                        Good luck 🙂

                        1 Reply Last reply Reply Quote 1
                        • 1 / 1
                        1 out of 11
                        • First post
                          1/11
                          Last post

                        Got a burning SEO question?

                        Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                        Start my free trial


                        Browse Questions

                        Explore more categories

                        • Moz Tools

                          Chat with the community about the Moz tools.

                        • SEO Tactics

                          Discuss the SEO process with fellow marketers

                        • Community

                          Discuss industry events, jobs, and news!

                        • Digital Marketing

                          Chat about tactics outside of SEO

                        • Research & Trends

                          Dive into research and trends in the search industry.

                        • Support

                          Connect on product support and feature requests.

                        • See all categories

                        Related Questions

                        • AlisonMills

                          Desktop & Mobile XML Sitemap Submitted But Only Desktop Sitemap Indexed On Google Search Console

                          Hi! The Problem We have submitted to GSC a sitemap index. Within that index there are 4 XML Sitemaps. Including one for the desktop site and one for the mobile site. The desktop sitemap has 3300 URLs, of which Google has indexed (according to GSC) 3,000 (approx). The mobile sitemap has 1,000 URLs of which Google has indexed 74 of them. The pages are crawlable, the site structure is logical. And performing a Landing Page URL search (showing only Google/Organic source/medium) on Google Analytics I can see that hundreds of those mobile URLs are being landed on. A search on mobile for a longtail keyword from a (randomly selected) page shows a result in the SERPs for the mobile page that judging by GSC has not been indexed. Could this be because we have recently added rel=alternate tags on our desktop pages (and of course corresponding canonical ones on mobile). Would Google then 'not index' rel=alternate page versions? Thanks for any input on this one. PmHmG

                          Technical SEO | Dec 21, 2017, 12:29 AM | AlisonMills
                          0
                        • thinkLukeSEO

                          Backlinks that go to a redirected URL

                          Hey guys, just wondering, my client has 3 websites, 2 of 3 will be closed down and the domains will be permanently redirected to the 1 primary domain - however they have some high quality backlinks pointing the domains that will be redirected. How does this effective SEO? Domain One (primary - getting redesign and rebuilt) - not many backlinks
                          Domain Two (will redirect to Domain One) - has quality backlinks
                          Domain Three (will redirect to Domain One) - has quality backlinks When the new website is launched on Domain One I will contact the backlink providers and request they update their URL - i assume that would be the best.

                          Technical SEO | Feb 21, 2023, 1:08 PM | thinkLukeSEO
                          0
                        • 94501

                          URL has caps, but canonical does not. Now what?

                          Hi, Just started working with a site that has the occasional url with a capital, but then the url in the canonical as lower case. Neither, when entered in a browser, resolves to the other. It's a Shopify site. What do you think I should do?

                          Technical SEO | Feb 21, 2017, 10:51 AM | 94501
                          0
                        • yacpro13

                          Resubmit sitemaps on every change?

                          Hello Mozers, Our sitemaps were submitted to Google and Bing, and are successfully indexed. Every time pages are added to our store (ecommerce), we re-generate the xml sitemap. My question is: should we be resubmitting the sitemaps every time their content change, or since they were submitted once can we assume that the crawlers will re-download the sitemaps by themselves (I don't like to assume). What are best practices here? Thanks!

                          Technical SEO | Apr 13, 2016, 1:28 PM | yacpro13
                          1
                        • wcbuckner

                          Best way to noindex long dynamic urls?

                          I just got a Mozcrawl back and see lots of errors for overly dynamic urls. The site is a villa rental site that gives users the ability to search by bedroom, amenities, price, etc, so I'm wondering what the best way to keep these types of dynamically generated pages with urls like /property-search-page/?location=any&status=any&type=any&bedrooms=9&bathrooms=any&min-price=any&max-price=any from indexing. Any assistance will be greatly appreciated : )

                          Technical SEO | Jan 29, 2014, 2:46 PM | wcbuckner
                          0
                        • bonnierSEO

                          Special characters in URL

                          Will registered trademark symbol within a URL be bad? I know some special characters are unsafe (#, >, etc.) but can not find anything that mentions registered trademark. Thanks!

                          Technical SEO | Dec 10, 2013, 9:54 PM | bonnierSEO
                          0
                        • DMGoo

                          HTML Sitemap Pagination?

                          Im creating an a to z type directory of internal pages within a site of mine however there are cases where there are over 500 links within the pages. I intend to use pagination (rel=next/prev) to avoid too many links on the page but am worried about indexation issues. should I be worried?"

                          Technical SEO | Sep 12, 2012, 8:47 PM | DMGoo
                          0
                        • rakesh_patel

                          Products with discrete URLs for each color

                          here is the issue.  i have an ecommerce site that on a category page, shows each individual color for each product sold.  and there is a distinct URL for each color.  each product page shares the same content, with the only potentially differentiating factor being customer reviews (not nearly enough of these to differentiate anything). so we have URLs like: www.domain.com/product-green www.domain.com/product-yellow www.domain.com/product-red and so on. i am looking for a way to consolidate these URL while still showing all colors on the category page.  the first solution i am considering is using the hash tag.  so we would create www.domain.com/product#green, www.domain.com/product#yellow, www.domain.com/product#red.  if possible, i would set the canonical tag as www.domain.com/product. the second solution would be to use the canonical tag and keep the URLs as is.  the issue i see here is that we would need to create www.domain.com/product and show that page somewhere.  www.domain.com/product would the URL that the above color URLs would canonicalize to. what would be the preferred solution?  or is there something else?

                          Technical SEO | Sep 14, 2011, 6:58 PM | rakesh_patel
                          0

                        Get started with Moz Pro!

                        Unlock the power of advanced SEO tools and data-driven insights.

                        Start my free trial
                        Products
                        • Moz Pro
                        • Moz Local
                        • Moz API
                        • Moz Data
                        • STAT
                        • Product Updates
                        Moz Solutions
                        • SMB Solutions
                        • Agency Solutions
                        • Enterprise Solutions
                        • Digital Marketers
                        Free SEO Tools
                        • Domain Authority Checker
                        • Link Explorer
                        • Keyword Explorer
                        • Competitive Research
                        • Brand Authority Checker
                        • Local Citation Checker
                        • MozBar Extension
                        • MozCast
                        Resources
                        • Blog
                        • SEO Learning Center
                        • Help Hub
                        • Beginner's Guide to SEO
                        • How-to Guides
                        • Moz Academy
                        • API Docs
                        About Moz
                        • About
                        • Team
                        • Careers
                        • Contact
                        Why Moz
                        • Case Studies
                        • Testimonials
                        Get Involved
                        • Become an Affiliate
                        • MozCon
                        • Webinars
                        • Practical Marketer Series
                        • MozPod
                        Connect with us

                        Contact the Help team

                        Join our newsletter
                        Moz logo
                        © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                        • Accessibility
                        • Terms of Use
                        • Privacy

                        Looks like your connection to Moz was lost, please wait while we try to reconnect.