undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Technical SEO
  4. 2 sitemaps on my robots.txt?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

2 sitemaps on my robots.txt?

Technical SEO
4
6
12.6k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • Webicultors
    Webicultors last edited by Jan 14, 2016, 8:00 AM

    Hi,

    I thought that I just could link one sitemap from my site's robots.txt but... I may be wrong.

    So, I need to confirm if this kind of implementation is right or wrong:

    robots.txt for Magento Community and Enterprise

    ...
    Sitemap: http://www.mysite.es/media/sitemap/es.xml
    Sitemap: http://www.mysite.pt/media/sitemap/pt.xml

    Thanks in advance,

    1 Reply Last reply Reply Quote 0
    • MonkStein
      MonkStein last edited by Nov 30, 2016, 6:17 PM Nov 30, 2016, 6:17 PM

      We recently changed our protocol to https

      We have in our robots.txt our new https sitemap link

      Our agency is recommending we add another sitemap in our robots.txt file to our insecure sitemap - while google is reindexing our secure protocol. They recommend this as a way for all SEs to pick up on 301 redirects and swap out unsecured results in the index more efficiently.

      Do you agree with this?

      I am in the camp that we should have have our https sitemap and google will figure it out and having 2 sitemaps one to our old http and one to our new https in our robots.txt is redundant and may be viewed as duplicate content, not as a positive of helping SEs to see 301s better to reindex secure links.

      Whats your thought? Let me know if I need to explain more.

      1 Reply Last reply Reply Quote 0
      • topic:timeago_earlier,11 months
      • Mobilio
        Mobilio @Webicultors last edited by Jan 14, 2016, 9:19 AM Jan 14, 2016, 9:19 AM

        Well if both sitemaps are for same site then it's OK. But it's much better to implement hreflang as this is explained here:https://support.google.com/webmasters/answer/2620865?hl=en

        I'm not sure that Magento can do this but you always can hire 3rd party dev for building plugin/module for this.

        1 Reply Last reply Reply Quote 1
        • Webicultors
          Webicultors last edited by Jan 14, 2016, 9:12 AM Jan 14, 2016, 9:12 AM

          ok, just one detail: these domains are for a multilang site.

          I mean, both have quite the same content: one in spanish and the other un portuguese.

          Thanks a lot.

          Mobilio 1 Reply Last reply Jan 14, 2016, 9:19 AM Reply Quote 2
          • Mobilio
            Mobilio last edited by Jan 14, 2016, 9:08 AM Jan 14, 2016, 9:08 AM

            You can also have multiple sitemaps on 3rd sites. Look at Moz robots.txt:
            Sitemap: https://moz.com/blog-sitemap.xml
            Sitemap: https://moz.com/ugc-sitemap.xml
            Sitemap: https://moz.com/profiles-sitemap.xml
            Sitemap: http://d2eeipcrcdle6.cloudfront.net/past-videos.xml
            Sitemap: http://app.wistia.com/sitemaps/36357.xml

            Also Google.com robots.txt:
            Sitemap: http://www.gstatic.com/culturalinstitute/sitemaps/www_google_com_culturalinstitute/sitemap-index.xml
            Sitemap: http://www.gstatic.com/dictionary/static/sitemaps/sitemap_index.xml
            Sitemap: http://www.gstatic.com/earth/gallery/sitemaps/sitemap.xml
            Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
            Sitemap: http://www.gstatic.com/trends/websites/sitemaps/sitemapindex.xml
            Sitemap: https://www.google.com/sitemap.xml

            Also Bing.com robots.txt:
            Sitemap: http://cn.bing.com/dict/sitemap-index.xml
            Sitemap: http://www.bing.com/offers/sitemap.xml

            So using multiple sitemaps it's OK and they can be also hosted on 3rd party server.

            1 Reply Last reply Reply Quote 1
            • donford
              donford last edited by Jan 14, 2016, 8:58 AM Jan 14, 2016, 8:58 AM

              Hello,

              Yes, multiple sitemaps are okay, and sometimes even advised!

              You can read Google's official response here."..it's fine for multiple Sitemaps to live in the same directory (as many as you want!)..."

              And you can see a case study showing how multiple sitemaps has helped traffic here on Moz.

              Hope this helps,

              Don

              1 Reply Last reply Reply Quote 1
              • 1 / 1
              1 out of 6
              • First post
                1/6
                Last post

              Got a burning SEO question?

              Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


              Start my free trial


              Browse Questions

              Explore more categories

              • Moz Tools

                Chat with the community about the Moz tools.

              • SEO Tactics

                Discuss the SEO process with fellow marketers

              • Community

                Discuss industry events, jobs, and news!

              • Digital Marketing

                Chat about tactics outside of SEO

              • Research & Trends

                Dive into research and trends in the search industry.

              • Support

                Connect on product support and feature requests.

              • See all categories

              Related Questions

              • rijwielcashencarry040

                Good robots txt for magento

                Dear Communtiy, I am trying to improve the SEO ratings for my website www.rijwielcashencarry.nl (magento). My next step will be implementing robots txt to exclude some crawling pages.
                Does anybody have a good magento robots txt for me? And what need i copy exactly? Thanks everybody! Greetings, Bob

                Technical SEO | Jan 13, 2017, 3:29 PM | rijwielcashencarry040
                0
              • kcb8178

                Is there a limit to how many URLs you can put in a robots.txt file?

                We have a site that has way too many urls caused by our crawlable faceted navigation.  We are trying to purge 90% of our urls from the indexes.  We put no index tags on the url combinations that we do no want indexed anymore, but it is taking google way too long to find the no index tags.  Meanwhile we are getting hit with excessive url warnings and have been it by Panda. Would it help speed the process of purging urls if we added the urls to the robots.txt file?  Could this cause any issues for us?  Could it have the opposite effect and block the crawler from finding the urls, but not purge them from the index? The list could be in excess of 100MM urls.

                Technical SEO | Oct 23, 2015, 4:40 AM | kcb8178
                0
              • Mark_Ginsberg

                Blocking Affiliate Links via robots.txt

                Hi, I work with a client who has a large affiliate network pointing to their domain which is a large part of their inbound marketing strategy. All of these links point to a subdomain of affiliates.example.com, which then redirects the links through a 301 redirect to the relevant target page for the link. These links have been showing up in Webmaster Tools as top linking domains and also in the latest downloaded links reports. To follow guidelines and ensure that these links aren't counted by Google for either positive or negative impact on the site, we have added a block on the robots.txt of the affiliates.example.com subdomain, blocking search engines from crawling the full subddomain. The robots.txt file is the following code: User-agent: * Disallow: / We have authenticated the subdomain with Google Webmaster Tools and made certain that Google can reach and read the robots.txt file. We know they are being blocked from reading the affiliates subdomain. However, we added this affiliates subdomain block a few weeks ago to the robots.txt, but links are still showing up in the latest downloads report as first being discovered after we added the block. It's been a few weeks already, and we want to make sure that the block was implemented properly and that these links aren't being used to negatively impact the site. Any suggestions or clarification would be helpful - if the subdomain is being blocked for the search engines, why are the search engines following the links and reporting them in the www.example.com subdomain GWMT account as latest links. And if the block is implemented properly, will the total number of links pointing to our site  as reported in the links to your site section be reduced, or does this not have an impact on that figure?From a development standpoint, it's a much easier fix for us to adjust the robots.txt file than to change the affiliate linking connection from a 301 to a 302, which is why we decided to go with this option.Any help you can offer will be greatly appreciated.Thanks,Mark

                Technical SEO | Nov 19, 2013, 7:37 PM | Mark_Ginsberg
                0
              • Saijo.George

                Should all pagination pages be included in sitemaps

                How important is it for a sitemap to include all individual urls for the paginated content. Assuming the rel next and prev tags are set up would it be ok to just have the page 1 in the sitemap ?

                Technical SEO | Nov 15, 2013, 3:44 PM | Saijo.George
                0
              • JorgeUmana

                Is there a suggested limit to the amount of links on a sitemap?

                Currently, I have an error on my moz dashboard indicating there are too many links on one of my pages.  That page is the sitemap.  It was my understanding all internal pages should be linked to the sitemap. Can any mozzers help clarify the best practice here? Thanks, Clayton

                Technical SEO | Dec 11, 2012, 7:36 PM | JorgeUmana
                0
              • tanlup

                Host sitemaps on S3?

                Hey guys, I run a dynamic web service and I will start building static sitemaps for it pretty soon. The fact that my app lives in a multitude of servers doesn't make it easy to distribute frequently updated static files throughout the servers. My idea was to host the files in AWS S3 and point my robots.txt sitemap directive there. I'll use a sitemap index so, every other sitemap will be hosted on S3 as well. I could dynamically mirror the content from the files in S3 through my app, but that would be a little more resource intensive than just serving the static files from a common place. Any ideas? Thanks!

                Technical SEO | Dec 5, 2012, 7:15 PM | tanlup
                0
              • TalkInThePark

                Googlebot does not obey robots.txt disallow

                Hi Mozzers! We are trying to get Googlebot to steer away from our internal search results pages by adding a parameter "nocrawl=1" to facet/filter links and then robots.txt disallow all URLs containing that parameter. We implemented this late august and since that, the GWMT message "Googlebot found an extremely high number of URLs on your site", stopped coming. But today we received yet another. The weird thing is that Google gives many of our nowadays robots.txt disallowed URLs as examples of URLs that may cause us problems. What could be the reason? Best regards, Martin

                Technical SEO | Nov 29, 2012, 3:52 AM | TalkInThePark
                0
              • kchandler

                Robots.txt File Redirects to Home Page

                I've been doing some site analysis for a new SEO client and it has been brought to my attention that their robots.txt file redirects to their homepage. I was wondering: Is there a benfit to setup your robots.txt file to do this? Will this effect how their site will get indexed? Thanks for your response! Kyle Site URL: http://www.radisphere.net/

                Technical SEO | Mar 18, 2011, 11:01 AM | kchandler
                0

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy

              Looks like your connection to Moz was lost, please wait while we try to reconnect.