Do I need a separate robots.txt file for my shop subdomain?
-
Hello Mozzers!
Apologies if this question has been asked before, but I couldn't find an answer so here goes...
Currently I have one robots.txt file hosted at https://www.mysitename.org.uk/robots.txt
We host our shop on a separate subdomain https://shop.mysitename.org.uk
Do I need a separate robots.txt file for my subdomain? (Some Google searches are telling me yes and some no and I've become awfully confused!
-
Thank you. I want to disallow specific URLs on the subdomain and add the shop sitemap in the robots.txt file. So I'll go ahead and create another!
-
You go be fine without one. You only need one if you want to manage that subdmain: add specific xml sitemaps links in robots.txt, cut access to specific folders for that subdomain.
if you don't need any of that - just move forward without one.
-
Currently we just have: User-agent: *
I'm in the process of optimising.
-
It depends what currently is in your robots.txt. Usually it would be useful to have another one for your subdomain.
-
Yes, I would have a seperate robots.txt files.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Utilizing one robots.txt for two sites
I have two sites that are facilitated hosting in similar CMS. Maybe than having two separate robots.txt records (one for every space), my web office has made one which records the sitemaps for the two sites, similar to this:
Technical SEO | | eulabrant0 -
I have two robots.txt pages for www and non-www version. Will that be a problem?
There are two robots.txt pages. One for www version and another for non-www version though I have moved to the non-www version.
Technical SEO | | ramb0 -
Href lang issues - help needed!
Hi, I have an issue with Google indexing the US version of our website rather than the UK version on Google.co.uk. I have added hreflang tags to both sites (https://www.pacapod.com/ and https://us.pacapod.com/), have updated and submitted an XML sitemap for each website and checked that the country targeting in search console is set-up correctly but Google are still indexing the wrong website. I would be grateful for any assistance with this issue. Many thanks Eddie
Technical SEO | | mypetgiftbox0 -
Is sitemap required on my robots.txt?
Hi, I know that linking your sitemap from your robots.txt file is a good practice. Ok, but... may I just send my sitemap to search console and forget about adding ti to my robots.txt? That's my situation: 1 multilang platform which means... ... 2 set of pages. One for each lang, of course But my CMS (magento) only allows me to have 1 robots.txt file So, again: may I have a robots.txt file woth no sitemap AND not suffering any potential SEO loss? Thanks in advance, Juan Vicente Mañanas Abad
Technical SEO | | Webicultors0 -
CPanel Redirects: Know How Needed
I have ran into something I have not ran into in the past. cpanel, setting a 301 redirect. --Drop down domain choice ----- / ---- ?attachment_id=539 http://ipromotis.com (Only box marked is redirect with or without www) Produces 301 Redirect http://ipromotis.com to http://ipromotis.com
Technical SEO | | yeagerd0 -
Does Bing ignore robots txt files?
Bonjour from "Its a miracle is not raining" Wetherby Uk 🙂 Ok here goes... Why despite a robots text file excluding indexing to site http://lewispr.netconstruct-preview.co.uk/ is the site url being indexed in Bing bit not Google? Does bing ignore robots text files or is there something missing from http://lewispr.netconstruct-preview.co.uk/robots.txt I need to add to stop bing indexing a preview site as illustrated below. http://i216.photobucket.com/albums/cc53/zymurgy_bucket/preview-bing-indexed.jpg Any insights welcome 🙂
Technical SEO | | Nightwing0 -
.lbi file - SEO friendly or not?
Up until yesterday afternoon i had never heard of a .lbi file. It turns out it is a library file used by Adobe Dreamweaver. From what i can tell it works like a client side included but i am unsure of the technology behind it. The issue:
Technical SEO | | kchandler
When running through a recent SEO audit for a new client i found these .lbi files being used all over there site for site wide callouts and even navigation. When viewing this content through firebug or in the browser you can see the executed HTML content but when viewing the source or the page in seo-browser.com the content is nowhere to be seen. So my thought is this is not SEO friendly and is the same as displaying content in any client-side script like JavaScript or JQuery. Any feedback or thoughts on this subject would be awesome, especially if anyone has used these previously. Unfortunately i cannot share the client site but i would be more than happy to answer any questions if more detail is needed. Thanks in advance - Kyle0 -
Page Analysis Difference Between Root and Subdomain
I have a site where the canonical version is the subdomain www, with a permanent redirect to ensure this is so. When I do a page analysis from the MozBar for the domain I see that www and *.domain are both displayed, with numbers from *.domain being shown by default in the mozbar. Does MozBar show *.domain numbers by default, and do I correctly understand that the (higher) www numbers displayed in page analysis for www are valid and a result of my canonical strategy?
Technical SEO | | waynekolenchuk0