Help with Robots.txt On a Shared Root
-
Hi,
I posted a similar question last week asking about subdomains but a couple of complications have arisen.
Two different websites I am looking after share the same root domain which means that they will have to share the same robots.txt. Does anybody have suggestions to separate the two on the same file without complications? It's a tricky one.
Thank you in advance.
-
Okay so if you have one root domain you can only have one robots.txt file.
The reason I asked for an example is in the case there was something you could put in the robots.txt to differentiate the two.
For example if you have
thisdomain.com and thatdomain.com
However if "thatdomain.com" uses a folder called shop ("thatdomain.com/shop") than you could prefix all your robots.txt file entries with /shop provided that "thisdomain.com" doesn't use the folder shop, Then all the /shop entries would only be applicable to "thatdomain.com". Does this make sense?
Don
-
It's not so much that one is a subdomain, it's that they are as different as Google and Yahoo yet they share the same root. I wish I could show you but I can't because of confidentiality.
The 303 wasn't put in place by me, I would have strongly suggested another method. I think it was set up so that both websites could be controlled from the same login but it's opened a can of worms for SEO.
I don't want the two separate robots files, the developer insists it has to be that way.
-
Can you provide me an example of the way the domains look... Specifically where the root pages are.
Additionally, if you are redirecting 303 one of the domains to the other why do you want two different robots.txt files? The one being 303 will always redirect to the other...?
Depending on the structures you can create one robots.txt file that deals with 2 different domains provided there is something unique about the root folders.
-
Thanks for your help so far.
The two different websites are different name domains but share the same root as it's been built this way on Typo3. I don't know of the developer's justification for the 303, it's something I wish we could change.
I'm not sure if there are specific tags you can put in the sole robots.txt to differentiate the two, have read a few conflicting arguments about how to do it.
-
Okay so if you're using a 303 then you're saying the content you want for X site is actually located at Y site.Which means you do not have 2 different sub domains. So there is no need for 2 robots.txt files and your developer is correct you can't use 2 robots.txt files. Since one site would be pointing to the other you only have one sub-domain.
However, 303 is in general a poor way to use a redirect and likely should be 301.. but I would have to understand why the 303 is being used to say that with 100% certainty. See a quick article about 303 here..
Hope this answers the question,
Don
-
It's Fasthosts. The developer is certain that we can't use the two separate robots files. The second website has been set up on a 303.
-
What host are you using?
-
The developer of the website insists that they have to share the same robots.txt, I am really not sure how he's set it up this way. I am beyond befuddled with this!
-
The subdomain has to be separated from the root in some fashion. I would assume depending on your host that there is a separate folder for the subdomain stuff. Otherwise it would be chaos. Say you installed forums on your forum subdomain and a e-commerce on your shop subdomain... which index.php page would be served?
There has to be some separation, review your file manager and look for the sub-domain folders. Once found you simply put a robots.txt into each of those folders.
Hope this helps,
Don
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Looking for someone to help with a delisted site after a malicious hack
Hello, My website - www.musillawfirm.com was recently hacked and has been de-listed by google. It had some sort of a crypto mining script on it that I was able to remove. It shows up if you type in the domain but even a generic search for "musil law firm" does not show the site - it used to rank # 1 for that term and #1 or 2 for immigration lawyer in my local area. If anyone can assist me in getting it re-indexed please let me know and let me know how much it would cost. I tried getting it re-indexed through the search console, but no luck. Thank you kindly
Intermediate & Advanced SEO | | musillawfirm0 -
Blank Cart Pages Showing as Duplicate, HELP
Hi Everyone, I'm seeing a bunch of URLs that look something like this [ domain.com/cart?add&id_product=42&token=776d4a08721f3d8c920e287248797547] showing as duplicate content in my Moz crawls. I think these are just blank pages for the most part. Is there anything to be concerned with here? Is there a way to clean this up? Thanks! Ricky
Intermediate & Advanced SEO | | RickyShockley0 -
Help! Website Page Structure.
Hi there, I have a cupcake website; www.cupcakesdelivered.com.au To date, we have sold only regular cupcakes. Moving forward, we are about to start selling lots of different sorts of cupcakes and want to categorise them - i.e.; sport cupcakes, corporate cupcakes, movie-themed cupcakes etc. I am looking for a recommendation on how best to structure this in terms of pages / domains / subdomains etc, so as to best support SEO. Your help would be greatly appreciated!! Thank you, Laura.
Intermediate & Advanced SEO | | cupcakesdelivered0 -
Will Schema help my website?
I'm doing SEO on a website, zing.co.nz, which is a soon to launch company. At the moment there is a splash sight up, which will be replaced by the real sight in a few weeks upon launch. Is it worth me putting in Schemas (for the first time) so that it is recognized as an organization? Will this effect us in the serps? Thanks for your help 🙂
Intermediate & Advanced SEO | | Startupfactory0 -
Help - Lost Ranking - What did I screw up?!
Hi, We're working with a local service provider with a location specific keyword (not a real example: "orlando plumbing contractors"). Background:
Intermediate & Advanced SEO | | AaronHenry
In recent history the client updated a new site design and upgraded from Joomla 1.5 to Joomla 2.5. Of course there were duplicate content issues which have been resolved with the help of AceSEF. Duplicate content, title tags, and other content issues are handled as soon as they appear in GWT or MOZ. Additionally, a high number of backlinks were lost when the latest Google update hit. Many of these sites were of sites that no longer existed or were spammy and flushed out. Some were lost due to the previous SEO firm literally removing backlinks and switching them to their new client (seo firm was putting all of the work the client paid for under their name to control everything). Current Situation: The backlink loss seems to have been stopped (hopefully) because we are using a new strategy that relies solely on the quality of the links, surrounding text, varied anchor text, relevancy, etc.) However, we tried an experiment on just one of the clients keywords. That experiment seems to have blown up in our faces evidently. The landing page for the location specific keyword has dropped from the index completely (it seems), but only when searching broad. When using exact match with quotes like the example quoted above ("orlando plumbing contractors") the landing page appears, but several ranks lower. We were ranked yesterday 6/23/13, but as of today 6/24/13 are no longer ranked. On broad matches, non-relevant sites and even a site that shows only a broken server configuration is outranking the client (they appear for the broad search, but the client does not). What Was Done
We recently created a press release for and posted it on a press release site. We then created a link back to the landing page (exact match anchor text). We posted the PR article to several social sites (Google plus, folkd, delicious, twitter, stumble upon, diigo). We also created a blog article (on-site) on site for that, creating links back to the landing page (the links all had exact anchor text). We posted that blog article to social news sites (facebook, stumble, delicious) and included a ping. The PR article was manually rewritten and posted to the PR site (we had to make 2 versions of the PR; one for the blog and one for the PR site). The Result
The client ending up dropping off the broad search rankings, but only slipped a few for "exact match" (with quotes). The PR article that was created is now ranked on page 3 for the board keyword and is still beat by non working sites. We suspect that the exact anchor text could be causing this problem. Anyone else have an idea (we're scratching our heads and trying not to freak out at the same time).0 -
Crazy long weird URLs... help
I have a HTML website, mysite1.com, and I placed a link on the home page to another one of my sites, mysite2.com Today I checked the links to mysite2.com in Majestic and noticed 24 links coming from the mysite1.com instead of just one link. The URLs from mysite1.com that are showing in Majestic are like this mysite1.com/?epl=4donafvFK3fMXxZXMWQRQLodmPchoXCK5C7-kbBv_agkwlkJrZAoaSDVUlhqFmUqt0f8c2Q6jF6GO6DNMnbidqRsikriF-IEBEt5okmICLEB0FxP36GrsxoPGQ3SGBo1PVR7itDUA4CYmjypn5gi mysite1.com,was inherited from a friend and I believe that it was originally built in Frontpage. Can you tell me how I can get rid of these multiple links as I only want 1 showing from the home page Thanks in advance
Intermediate & Advanced SEO | | JohnPeters0 -
Help needed on Google Webmaster tools
Hi I notice that one of my oldest sites , even if I put hundred of backlinks (good or bad) google webmaster tools never index more like 20 per day. Why is this happening? They control it? I mean they dont let them all to get indexed and they take it slowly slowly? If I put just 20 per day is the ideal link building amount? Thnk you
Intermediate & Advanced SEO | | nyanainc0 -
Do Share buttons take LinkJuice?
Hi, I'm using AddThis sharing on my site. The implementation is by embedding a href's all over the site. Do these buttons take my LinkJuice? Thanks
Intermediate & Advanced SEO | | BeytzNet0