Utilizing one robots.txt for two sites
-
I have two sites that are facilitated hosting in similar CMS. Maybe than having two separate robots.txt records (one for every space), my web office has made one which records the sitemaps for the two sites, similar to this:
-
You can use the same syntax on both root domains but realistically you can't have 1 robots.txt point to something else (with the exception of XML sitemaps). The only option that I see here is that you have one file that gets served from 1 server on both requests from two sites but that seems like a lot of engineering overkill for a very simple problem (just duplicating files).
-
@eulabrant It is advisable to use separate robots.txt files for different properties.
If there is a subdomain, you can utilize the main parent domain robots.txt file for the corresponding child subdomain.
-
Hey @eulabrant You didn't really ask a question here but I'll assume you want to know if you can use one robots.txt for two sites?
You could duplicate it but you can't really use it for both. I also assume you are missing something off this post as you are referring to "this:" but not that 'this' is
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to stop robots.txt restricting access to sitemap?
I'm working on a site right now and having an issue with the robots.txt file restricting access to the sitemap - with no web dev to help, I'm wondering how I can fix the issue myself? The robots.txt page shows User-agent: * Disallow: / And then sitemap: with the correct sitemap link
Technical SEO | | Ad-Rank0 -
"Moz encountered an error on one or more pages on your site" Error
I have been receiving this error for a while: "Moz encountered an error on one or more pages on your site" It's a Multi-Lingual Wordpress website, the robots.txt is set to allow crawlers on all links and I have followed the same process for other website I've done yet I'm receiving this error for this site.
Technical SEO | | JustinZimri0 -
Google Site Search
I'm considering to implement google site search bar into my site.
Technical SEO | | JonsonSwartz
I think I probably choose the version without the ads (I'll pay for it). does anyone use Google Site Search and can tell if it's a good thing? does it affects in any way on seo? thank you0 -
Why does my site rank so badly
its my turn to ask the interminable question why does my site rank so badly? site is: marriagerecords.org.uk. it was #1 for 'marriage records' on google for about 6 months. then it was 5th to 10th for about 2 months. now it is nowhere for this phrase and anything else, none of the pages I have written rank for anything. I have spent hours upon hours researching original content and I have got some great backlinks from sites like wrexham.gov.uk and somerset.gov.uk (some dont show in opensiteexplorer yet). im guessing im over-optimizing something but i'd love some concrete fixes if anyone could suggest any. thanks, tom
Technical SEO | | lethal0r0 -
More than one web domain
Hi What is best when considering using more than one domain on a website, what the best policy ? it's a question I get asked a lot, usually because prior to any seo efforts the main domain name purchased is not keyword rich or an abbreviated company name etc. what impact do new domains have on SEO compared to older existing domains- is it worth changing a generic company named domain for a keyword rich domain? if having multiple domains pointing to the site is beneficial how best is it to configure? How do i inform Google? How do both domains get index when there is only one physical site? How should i monitor it, Analytic s, SEO moz? Will the domain compete against each other and effect SEO rank? Are they best used for marketing purposes on external sites, adverts driving traffic to the main site. I'm aware there are lots of questions above, any answers/ opinions , links to further info would be greatly appreciated. cheers
Technical SEO | | Bristolweb0 -
Robots.txt and 301
Hi Mozzers, Can you answer something for me please. I have a client and they have 301 re-directed the homepage '/' to '/home.aspx'. Therefore all or most of the linkjuice is being passed which is great. They have also marked the '/' as nofollow / noindex in the Robots.txt file so its not being crawled. My question is if the '/' is being denied access to the robots is it still passing on the authority for the links that go into this page? It is a 301 and not 302 so it would work under normal circumstances but as the page is not being crawled do I need to change the Robots.txt to crawl the '/'? Thanks Bush
Technical SEO | | Bush_JSM0 -
Google.ca is showing our US site instead of our Canada Site
When our Canadian users who search on google.ca for our brand (e.g. Travelocity, Travelocity hotels, etc.), the first few results our from our US site (travelocity.com) rather than our Canadian site (travelocity.ca). In Google Webmaster Tools, we've adjusted the geotargeting settings to focus on the appropriate locale, but the wrong country TLD is still coming up at the top via google.ca. What's the best way to ensure our Canadian site comes up instead of the US site on google.ca? Thanks, Tory Smith
Technical SEO | | travelocitysearch
Travelocity0