Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • MozCon
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Technical SEO
    4. Multiple robots.txt files on server

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Multiple robots.txt files on server

    Technical SEO
    5
    7
    3720
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • mjukhud
      mjukhud last edited by

      Hi!

      I have previously hired a developer to put up my site and noticed afterwards that he did not know much about SEO. This lead me to starting to learn myself and applying some changes step by step.

      One of the things I am currently doing is inserting sitemap reference in robots.txt file (which was not there before). But just now when I wanted to upload the file via FTP to my server I found multiple ones - in different sizes - and I dont know what to do with them? Can I remove them? I have downloaded and opened them and they seem to be 2 textfiles and 2 dupplicates. Names:

      robots.txt (original dupplicate)
      robots.txt-Original (original)
      robots.txt-NEW (other content)
      robots.txt-Working (other content dupplicate)

      Would really appreciate help and expertise suggestions. Thanks!

      1 Reply Last reply Reply Quote 0
      • peakdistrictseo
        peakdistrictseo last edited by

        So what's the best policy if a site uses an e-commerce platform like Magento, which has a robots file, but also has a Wordpress blog installed to another folder. eg: /blog and uses a plugin like YOAST which generated a robots file of the Wordpress installation.

        Then you have 2 robots files, is this detrimental or no big deal?

        1 Reply Last reply Reply Quote 0
        • mjukhud
          mjukhud @seoman10 last edited by

          Thanks very much for the help!

          1 Reply Last reply Reply Quote 0
          • mjukhud
            mjukhud last edited by

            Thanks very much for the help!

            1 Reply Last reply Reply Quote 0
            • seoman10
              seoman10 last edited by

              Keep a backup and remove them.

              Search engines are only going to look at the file which is exactly called robots.txt variations of file name will be ignored.

              Do make sure the entries are correct in the main one though, you don't want Google crawling admin pages or other confidential areas of the site.

              mjukhud 1 Reply Last reply Reply Quote 1
              • mjukhud
                mjukhud @Mustansar last edited by

                Hi, thanks for the answer and help!

                Well, I only have one domain that has a webpage and no subdomains active (no blog-subdomain or similar) - so how can I configure that to the situation? Can I just remove all and upload the one I want, maybe?

                1 Reply Last reply Reply Quote 0
                • Mustansar
                  Mustansar last edited by

                  That's a good question, EMS.  The robots.txt protocol can get kind of 
                  confusing when you think about it too long, and it sounds like you've 
                  thought about this a bit.  However, in this case, it might help to 
                  look at robots.txt from the perspective of the spider.

                  When a spider finds a URL, it takes the whole domain name (everything 
                  between 'http://' and the next '/'), then sticks a '/robots.txt' on 
                  the end of it and looks for that file.  If that file exists, then the 
                  spider should read it to see where it is allowed to crawl.

                  In your case, Googlebot, or any other spider, should try to access 
                  three URLs: domainA.com/robots.txt, domainB.domainA.com/robots.txt, 
                  and domainB.com/robots.txt.  The rules in each are treated as 
                  separate, so disallowing robots from domainA.com/ should result in 
                  domainA.com/ being removed from search results while 
                  domainB.domainA.com/ remains unaffected, which does not sound like not 
                  something you want.

                  The problem you might have with the setup you have described is this-- 
                  in order to keep domainB.domainA.com out of the results, you would 
                  need to have domainB.domainA.com/robots.txt exclude robots, while 
                  domainB.com/robots.txt welcomes them.  This means that you would need 
                  to have a way to make domainB.domainA.com/ and domainB.com/ serve 
                  different information, and judging from what you've described, you 
                  have not set up your server to do so yet.

                  Of course, it is always possible that I have assumed to much about 
                  your situation, so it is a good idea to use Google's robots.txt 
                  analysis tool (see http://www.google.com/support/webmasters/bin/topic.py?topic=8475
                  ) to see if your robots.txt files already produce the results you 
                  want.

                  If using robots.txt files doesn't solve the problem, and assuming that 
                  you want to continue hosting all of your content on domainA.com, one 
                  strategy you really should look into would be setting up a 301 
                  redirect from the pages on domainB.domainA.com/ to domainB.com/ .  If 
                  you need more advice on how to do this with your server software, your 
                  hosting company's tech support would definitely be the best place to 
                  start, but this group is here to help if more isues arise. 🙂

                  Hope that helps!

                  mjukhud 1 Reply Last reply Reply Quote 0
                  • 1 / 1
                  • First post
                    Last post

                  Got a burning SEO question?

                  Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                  Start my free trial


                  Browse Questions

                  Explore more categories

                  • Moz Tools

                    Chat with the community about the Moz tools.

                  • SEO Tactics

                    Discuss the SEO process with fellow marketers

                  • Community

                    Discuss industry events, jobs, and news!

                  • Digital Marketing

                    Chat about tactics outside of SEO

                  • Research & Trends

                    Dive into research and trends in the search industry.

                  • Support

                    Connect on product support and feature requests.

                  • See all categories

                  Related Questions

                  • Leaf-a-mark

                    Redirect multiple domains to 1 domain or not?

                    Hi there, I have client who has multiple domains that already have some PA and DA. Problem is that most websites have the same content and rank better on different keywords.
                    I want to redirect all the websites to 1 domain because it’s easier to manage and it removes any duplicate content. Question is if I redirect domain x to domain y do the rankings of domain x increase on domain y? Or is it better to keep domain x separately to generate more referral traffic to domain y? Thanks in advance! Cheers

                    Technical SEO | | Leaf-a-mark
                    0
                  • AL123al

                    Umbrella company and multiple domains

                    I'm really sorry for asking this question yet again. I have searched through previous answers but couldn't see something exactly like this I think. There is a website called example .com. It is a sort of umbrella company for 4 other separate domains within it - 4 separate companies. The Home page of the "umbrella" company website is example.com. It is just an image with no content except navigation on it to direct to the 4 company websites. The other pages of website example.com are the 4 separate companies domains. So on the navigation bar there is : Home page = example.com company1page = company1domain.com company2page= company2domain.com etc. etc. Clicking "home" will take you back to example.com (which is just an image). How bad or good is this structure for SEO? Would you recommend any changes to help them rank better? The "home" page has no authority or links, and neither do 3 out of the 4 other domains. The 4 companies websites are independent in content (although theme is the same). What's bringing them altogether is under this umbrella website -  example.com. Thank you

                    Technical SEO | | AL123al
                    0
                  • ulefos

                    Remove html file extension and 301 redirects

                    Hi Recently I ask for some work done on my website from a company,  but I am not sure what they've done is right.
                    What I wanted was html file extensions to be removed like
                    /ash-logs.html to /ash-logs
                    also the index.html to www.timports.co.uk
                    I have done a crawl diagnostics and have duplicate page content and 32 page title duplicates. This is so doing my head in please help This is what is in the .htaccess file <ifmodule pagespeed_module="">ModPagespeed on
                    ModPagespeedEnableFilters extend_cache,combine_css, collapse_whitespace,move_css_to_head, remove_comments</ifmodule> <ifmodule mod_headers.c="">Header set Connection keep-alive</ifmodule> <ifmodule mod_rewrite.c="">Options +FollowSymLinks -MultiViews</ifmodule> DirectoryIndex index.html RewriteEngine On 
                     # Rewrite valid requests on .html files  RewriteCond %{REQUEST_FILENAME}.html -f RewriteRule ^ %{REQUEST_URI}.html?rw=1 [L,QSA] 
                     # Return 404 on direct requests against .html files RewriteCond %{REQUEST_URI} .html$  
                    RewriteCond %{QUERY_STRING} !rw=1 [NC]
                     RewriteRule ^ - [R=404] AddCharset UTF-8 .html # <filesmatch “.(js|css|html|htm|php|xml|swf|flv|ashx)$”="">#SetOutputFilter DEFLATE #</filesmatch> <ifmodule mod_expires.c="">ExpiresActive On
                    ExpiresByType image/gif "access plus 1 years"
                    ExpiresByType image/jpeg "access plus 1 years"
                    ExpiresByType image/png "access plus 1 years"
                    ExpiresByType image/x-icon "access plus 1 years"
                    ExpiresByType image/jpg "access plus 1 years"
                    ExpiresByType text/css "access 1 years"
                    ExpiresByType text/x-javascript "access 1 years"
                    ExpiresByType application/javascript "access 1 years"
                    ExpiresByType image/x-icon "access 1 years"</ifmodule> <files 403.shtml="">order allow,deny allow from all</files> redirect 301 /PRODUCTS http://www.timports.co.uk/kiln-dried-logs
                    redirect 301 /kindling_firewood.html http://www.timports.co.uk/kindling-firewood.html
                    redirect 301 /about_us.html http://www.timports.co.uk/about-us.html
                    redirect 301 /log_delivery.html http://www.timports.co.uk/log-delivery.html redirect 301 /oak_boards_delivery.html http://www.timports.co.uk/oak-boards-delivery.html
                    redirect 301 /un_edged_oak_boards.html http://www.timports.co.uk/un-edged-oak-boards.html
                    redirect 301 /wholesale_logs.html http://www.timports.co.uk/wholesale-logs.html redirect 301 /privacy_policy.html http://www.timports.co.uk/privacy-policy.html redirect 301 /payment_failed.html http://www.timports.co.uk/payment-failed.html redirect 301 /payment_info.html http://www.timports.co.uk/payment-info.html

                    Technical SEO | | ulefos
                    1
                  • AllIsWell

                    Multiple urls for posting multiple classified ads

                    Want to optimize referral traffic while at same time keep search engines happy and the ads posted. Have a client who advertises  on several classified ad sites around the globe. Which is better (post Panda), having multiple identical urls using canonicals  to redirect juice to original url? For example: www.bluewidgets.com is the original www.bluewidgetsusa.com www.blue-widgets-galore.com Or, should the duplicate pages be directed to original using a 301? Currently using duplicate urls. Am currently not using "nofollow" tags on those pages.

                    Technical SEO | | AllIsWell
                    0
                  • Webmaster123

                    I accidentally blocked Google with Robots.txt. What next?

                    Last week I uploaded my site and forgot to remove the robots.txt file with this text: User-agent: * Disallow: / I dropped from page 11 on my main keywords to past page 50. I caught it 2-3 days later and have now fixed it. I re-imported my site map with Webmaster Tools and I also did a Fetch as Google through Webmaster Tools. I tweeted out my URL to hopefully get Google to crawl it faster too. Webmaster Tools no longer says that the site is experiencing outages, but when I look at my blocked URLs it still says 249 are blocked. That's actually gone up since I made the fix. In the Google search results, it still no longer has my page title and the description still says "A description for this result is not available because of this site's robots.txt – learn more." How will this affect me long-term? When will I recover my rankings? Is there anything else I can do? Thanks for your input! www.decalsforthewall.com

                    Technical SEO | | Webmaster123
                    0
                  • IM_Learner

                    301 Redirect on a PDF, DOCX files?

                    Hi, I have to rename many pdf and docx files. How can I implement 301 redirect on them as they are linked from 'n' number of places? Regards, Shailendra Sial

                    Technical SEO | | IM_Learner
                    1
                  • LMDNYC

                    How should I structure a site with multiple addresses to optimize for local search??

                    Here's the setup: We have a website, www.laptopmd.com, and we're ranking quite well in our geographic target area. The site is chock-full of local keywords, has the address properly marked up, html5 and schema.org compliant, near the top of the page, etc. It's all working quite well, but we're looking to expand to two more locations, and we're terrified that adding more addresses and playing with our current set-up will wreak havoc with our local search results, which we quite frankly currently rock. My question is 1)when it comes time to doing sub-pages for the new locations, should we strip the location information from the main site and put up local pages for each location in subfolders? 1a) should we use subdomains instead of subfolders to keep Google from becoming confused? Should we consider simply starting identically branded pages for the individual locations and hope that exact-match location-based urls will make up for the hit for duplicate content and will overcome the difficulty of building a brand from multiple pages? I've tried to look for examples of businesses that have tried to do what we're doing, but all the advice has been about organic search, which i already have the answer to. I haven't been able to really find a good example of a small business with multiple locations AND good rankings for each location. Should this serve as a warning to me?

                    Technical SEO | | LMDNYC
                    0
                  • Grenadi

                    How to 301 multiple domain names to a single domain

                    Hey, I tried to find and answer to this seemingly simple question, but no luck. So, I have one domain name with a website attached to it. I also registered all the other domain names that are similar to it or have different extensions - I want to redirect all the other domain names to my one main domain name without getting penalised by the big G. It looks like this: www.mainsite.com - this is my main domain I also have www.mainsite.com.au, www.mainsite.org, and www.mainsite.org.au which I all want to just redirect to www.mainsite.com I have been told that the best way to do this is a 301 redirect, but to do that you need to make a CNAME for all the other domains that points to www.mainsite.com. My problem is that I cannot seem to create a CNAME record for http://mainsite.com - I have it working for http://www.mainsite.com but not the non www record. What should I be doing differently? Is it just my DNS provider is useless? Thanks, Anthony

                    Technical SEO | | Grenadi
                    0

                  Get started with Moz Pro!

                  Unlock the power of advanced SEO tools and data-driven insights.

                  Start my free trial
                  Products
                  • Moz Pro
                  • Moz Local
                  • Moz API
                  • Moz Data
                  • STAT
                  • Product Updates
                  Moz Solutions
                  • SMB Solutions
                  • Agency Solutions
                  • Enterprise Solutions
                  Free SEO Tools
                  • Domain Authority Checker
                  • Link Explorer
                  • Keyword Explorer
                  • Competitive Research
                  • Brand Authority Checker
                  • Local Citation Checker
                  • MozBar Extension
                  • MozCast
                  Resources
                  • Blog
                  • SEO Learning Center
                  • Help Hub
                  • Beginner's Guide to SEO
                  • How-to Guides
                  • Moz Academy
                  • API Docs
                  About Moz
                  • About
                  • Team
                  • Careers
                  • Contact
                  Why Moz
                  • Case Studies
                  • Testimonials
                  Get Involved
                  • Become an Affiliate
                  • MozCon
                  • Webinars
                  • Practical Marketer Series
                  • MozPod
                  Connect with us

                  Contact the Help team

                  Join our newsletter
                  Moz logo
                  © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                  • Accessibility
                  • Terms of Use
                  • Privacy

                  Looks like your connection to Moz was lost, please wait while we try to reconnect.