undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Technical SEO
  4. Multiple robots.txt files on server

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Multiple robots.txt files on server

Technical SEO
5
7
3.7k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • mjukhud
    mjukhud last edited by Sep 2, 2016, 6:28 AM

    Hi!

    I have previously hired a developer to put up my site and noticed afterwards that he did not know much about SEO. This lead me to starting to learn myself and applying some changes step by step.

    One of the things I am currently doing is inserting sitemap reference in robots.txt file (which was not there before). But just now when I wanted to upload the file via FTP to my server I found multiple ones - in different sizes - and I dont know what to do with them? Can I remove them? I have downloaded and opened them and they seem to be 2 textfiles and 2 dupplicates. Names:

    robots.txt (original dupplicate)
    robots.txt-Original (original)
    robots.txt-NEW (other content)
    robots.txt-Working (other content dupplicate)

    Would really appreciate help and expertise suggestions. Thanks!

    1 Reply Last reply Reply Quote 0
    • peakdistrictseo
      peakdistrictseo last edited by Jan 10, 2018, 8:06 AM Jan 10, 2018, 8:06 AM

      So what's the best policy if a site uses an e-commerce platform like Magento, which has a robots file, but also has a Wordpress blog installed to another folder. eg: /blog and uses a plugin like YOAST which generated a robots file of the Wordpress installation.

      Then you have 2 robots files, is this detrimental or no big deal?

      1 Reply Last reply Reply Quote 0
      • mjukhud
        mjukhud @seoman10 last edited by Sep 2, 2016, 10:39 AM Sep 2, 2016, 10:39 AM

        Thanks very much for the help!

        1 Reply Last reply Reply Quote 0
        • mjukhud
          mjukhud last edited by Sep 2, 2016, 10:39 AM Sep 2, 2016, 10:39 AM

          Thanks very much for the help!

          1 Reply Last reply Reply Quote 0
          • seoman10
            seoman10 last edited by Sep 2, 2016, 10:38 AM Sep 2, 2016, 10:28 AM

            Keep a backup and remove them.

            Search engines are only going to look at the file which is exactly called robots.txt variations of file name will be ignored.

            Do make sure the entries are correct in the main one though, you don't want Google crawling admin pages or other confidential areas of the site.

            mjukhud 1 Reply Last reply Sep 2, 2016, 10:39 AM Reply Quote 1
            • mjukhud
              mjukhud @Mustansar last edited by Sep 2, 2016, 7:03 AM Sep 2, 2016, 7:03 AM

              Hi, thanks for the answer and help!

              Well, I only have one domain that has a webpage and no subdomains active (no blog-subdomain or similar) - so how can I configure that to the situation? Can I just remove all and upload the one I want, maybe?

              1 Reply Last reply Reply Quote 0
              • Mustansar
                Mustansar last edited by Sep 2, 2016, 6:37 AM Sep 2, 2016, 6:37 AM

                That's a good question, EMS.  The robots.txt protocol can get kind of 
                confusing when you think about it too long, and it sounds like you've 
                thought about this a bit.  However, in this case, it might help to 
                look at robots.txt from the perspective of the spider.

                When a spider finds a URL, it takes the whole domain name (everything 
                between 'http://' and the next '/'), then sticks a '/robots.txt' on 
                the end of it and looks for that file.  If that file exists, then the 
                spider should read it to see where it is allowed to crawl.

                In your case, Googlebot, or any other spider, should try to access 
                three URLs: domainA.com/robots.txt, domainB.domainA.com/robots.txt, 
                and domainB.com/robots.txt.  The rules in each are treated as 
                separate, so disallowing robots from domainA.com/ should result in 
                domainA.com/ being removed from search results while 
                domainB.domainA.com/ remains unaffected, which does not sound like not 
                something you want.

                The problem you might have with the setup you have described is this-- 
                in order to keep domainB.domainA.com out of the results, you would 
                need to have domainB.domainA.com/robots.txt exclude robots, while 
                domainB.com/robots.txt welcomes them.  This means that you would need 
                to have a way to make domainB.domainA.com/ and domainB.com/ serve 
                different information, and judging from what you've described, you 
                have not set up your server to do so yet.

                Of course, it is always possible that I have assumed to much about 
                your situation, so it is a good idea to use Google's robots.txt 
                analysis tool (see http://www.google.com/support/webmasters/bin/topic.py?topic=8475
                ) to see if your robots.txt files already produce the results you 
                want.

                If using robots.txt files doesn't solve the problem, and assuming that 
                you want to continue hosting all of your content on domainA.com, one 
                strategy you really should look into would be setting up a 301 
                redirect from the pages on domainB.domainA.com/ to domainB.com/ .  If 
                you need more advice on how to do this with your server software, your 
                hosting company's tech support would definitely be the best place to 
                start, but this group is here to help if more isues arise. 🙂

                Hope that helps!

                mjukhud 1 Reply Last reply Sep 2, 2016, 7:03 AM Reply Quote 0
                • 1 / 1
                1 out of 7
                • First post
                  1/7
                  Last post

                Got a burning SEO question?

                Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                Start my free trial


                Browse Questions

                Explore more categories

                • Moz Tools

                  Chat with the community about the Moz tools.

                • SEO Tactics

                  Discuss the SEO process with fellow marketers

                • Community

                  Discuss industry events, jobs, and news!

                • Digital Marketing

                  Chat about tactics outside of SEO

                • Research & Trends

                  Dive into research and trends in the search industry.

                • Support

                  Connect on product support and feature requests.

                • See all categories

                Related Questions

                • Keith-007

                  Using GeoDNS across 3 server locations

                  Hi, I have multiple servers across UK and USA. I have a web site that serves both areas and was looking at cloning my sites and using GeoDNS to route visitors to the closest server to improve speed and experience So UK visitors would connect to UK dedicated server, North America - New York server and so on Is this a good way or would this effect SEO negatively. Cheers Keith

                  Technical SEO | Jul 1, 2020, 11:38 AM | Keith-007
                  1
                • Webicultors

                  2 sitemaps on my robots.txt?

                  Hi, I thought that I just could link one sitemap from my site's robots.txt but... I may be wrong. So, I need to confirm if this kind of implementation is right or wrong: robots.txt for Magento Community and Enterprise ...
                  Sitemap: http://www.mysite.es/media/sitemap/es.xml
                  Sitemap: http://www.mysite.pt/media/sitemap/pt.xml Thanks in advance,

                  Technical SEO | Nov 30, 2016, 6:17 PM | Webicultors
                  0
                • columbiaseo

                  Server is taking too long to respond - What does this mean?

                  A client has 3 sites that he would like for me to look at.  Whenever I attempt to on my home internet I get this message: The connection has timed out
                  The server is taking too long to respond. When I take my iphone off wifi and use AT&T, the site comes up fine.  What is going on here?

                  Technical SEO | Jul 26, 2013, 4:15 PM | columbiaseo
                  0
                • mkhGT

                  Are robots.txt wildcards still valid? If so, what is the proper syntax for setting this up?

                  I've got several URL's that I need to disallow in my robots.txt file. For example, I've got several documents that I don't want indexed and filters that are getting flagged as duplicate content. Rather than typing in thousands of URL's I was hoping that wildcards were still valid.

                  Technical SEO | Jul 9, 2013, 9:04 PM | mkhGT
                  0
                • ProjectLabs

                  Determining When to Break a Page Into Multiple Pages?

                  Suppose you have a page on your site that is a couple thousand words long. How would you determine when to split the page into two and are there any SEO advantages to doing this like being more focused on a specific topic. I noticed the Beginner's Guide to SEO is split into several pages, although it would concentrate the link juice if it was all on one page. Suppose you have a lot of comments. Is it better to move comments to a second page at a certain point? Sometimes the comments are not super focused on the topic of the page compared to the main text.

                  Technical SEO | Oct 18, 2012, 9:44 PM | ProjectLabs
                  1
                • jdossetti

                  Adding Rel Canonical to multiple pages

                  Hi, Our CMS generates a lot of duplicate content, (Different versions of every page for 3 different font sizes). There are many other reasons why we should drop this current CMS and go with something else, and we are in the process of doing that. But for now, does anyone know how would I do the following: I've created a spreadsheet that contains the following: Column 1: rel="canonical" tag for URL Column 2: Duplicate Content URL # 1 Column 3: Duplicate Content URL # 2 Column 4: Duplicate Content URL # 3 I want to add the tag from column 1 into the head of every page from column 2,3, and 4. What would be a fast way to do this considering that I have around 1800 rows. Check the screenshot of the builtwith.com result to see more information about the website if that helps. Farris bxySL

                  Technical SEO | Jul 20, 2012, 3:50 PM | jdossetti
                  0
                • ocelot

                  Duplicate content problem from an index.php file

                  Hi One of my sites is flagging a duplicate content problem which is affecting the search rankings. The duplicate problem is caused by http://www.mydomain.com/index.php which has a page rank of 26 How can I sort the duplicate content problem, as the main page should just be http://www.mydomain.com which has a page rank of 42 and is the stronger page with stronger links etc Many Thanks

                  Technical SEO | Jul 2, 2012, 5:40 PM | ocelot
                  0
                • neooptic

                  How to extract URLs from a site (without bringing the server down!)

                  Hi everybody. One of my clients is migrating to a new ecommerce platform, and we need to get a list of urls from the existing site to start mapping out the 301 redirects. Usually, I'd use a tool like Xenu or Integrity to crawl and output a list. However, the database and server setup is so bad that it can't handle the requests from these tools and it sends the site down. This, unsurprisingly, is one of the reasons for the migration. Does anybody know of a way to get a full list of urls without having to make a bunch of http requests which will kill the site? Any advice would be much appreciated!

                  Technical SEO | Nov 14, 2011, 7:03 PM | neooptic
                  0

                Get started with Moz Pro!

                Unlock the power of advanced SEO tools and data-driven insights.

                Start my free trial
                Products
                • Moz Pro
                • Moz Local
                • Moz API
                • Moz Data
                • STAT
                • Product Updates
                Moz Solutions
                • SMB Solutions
                • Agency Solutions
                • Enterprise Solutions
                Free SEO Tools
                • Domain Authority Checker
                • Link Explorer
                • Keyword Explorer
                • Competitive Research
                • Brand Authority Checker
                • Local Citation Checker
                • MozBar Extension
                • MozCast
                Resources
                • Blog
                • SEO Learning Center
                • Help Hub
                • Beginner's Guide to SEO
                • How-to Guides
                • Moz Academy
                • API Docs
                About Moz
                • About
                • Team
                • Careers
                • Contact
                Why Moz
                • Case Studies
                • Testimonials
                Get Involved
                • Become an Affiliate
                • MozCon
                • Webinars
                • Practical Marketer Series
                • MozPod
                Connect with us

                Contact the Help team

                Join our newsletter
                Moz logo
                © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                • Accessibility
                • Terms of Use
                • Privacy

                Looks like your connection to Moz was lost, please wait while we try to reconnect.