Do I need a separate robots.txt file for my shop subdomain?
-
Hello Mozzers!
Apologies if this question has been asked before, but I couldn't find an answer so here goes...
Currently I have one robots.txt file hosted at https://www.mysitename.org.uk/robots.txt
We host our shop on a separate subdomain https://shop.mysitename.org.uk
Do I need a separate robots.txt file for my subdomain? (Some Google searches are telling me yes and some no and I've become awfully confused!
-
Thank you. I want to disallow specific URLs on the subdomain and add the shop sitemap in the robots.txt file. So I'll go ahead and create another!
-
You go be fine without one. You only need one if you want to manage that subdmain: add specific xml sitemaps links in robots.txt, cut access to specific folders for that subdomain.
if you don't need any of that - just move forward without one.
-
Currently we just have: User-agent: *
I'm in the process of optimising.
-
It depends what currently is in your robots.txt. Usually it would be useful to have another one for your subdomain.
-
Yes, I would have a seperate robots.txt files.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does a subdomain benefit from being on a high authority domain?
I think the title sums up the question, but does a new subdomain get any ranking benefits from being on a pre-existing high authority domain. Or does the new subdomain have to fend for itself in the SERPs?
Technical SEO | | RG_SEO0 -
Speed benefits from loading images from a subdomain
I have read that loading images from a subdomain of your site instead of the main domain will give you speed benefits on load time. Has anyone actually seen that to be the case? Thanks!
Technical SEO | | Gordian0 -
Need Urgent Help
I have found one mistake that my place page address is little different than address on all local directories like on place page address is: 10010 S Tryon St #122 Charlotte, NC 28273 and on directories : 10010 South Tryon St 122 Charlotte, NC 28273 so on place page it is just "S" instead of South and "#" is before 122 but on all directories # is missing So what do you suggest ? Should i change address and re verify place page ? Re verify will put down place page value ???
Technical SEO | | mnkpso0 -
Directory Naming & File Organization
We're redoing an entire site and are going to reorganize, and link to the site's pages by directory instead of page name. So instead of:xyz.com/services/fixingtvs.phpit will be:xyz.com/fixingtvsAt first I was thinking 1 index.php page per directory but that will make content management really confusing with a bunch of files with the same name.Anyone have a better idea?Thanks,Matt
Technical SEO | | mattloht0 -
Domain targeting advice needed please
I would be interested in hearing the views of other seomozers on this issue please. I have a web server hosted in The Netherlands which I currently host my sites on, it is super fast (16core 24gb ram) and in 8 months has had 4 mins of downtime! On this server I wish to build a couple of ecommerce stores. However this is where my issue lays The first store I launch will be targeted at the UK market, however the domain I wish to use for it is a .com domain which has a moz ranting of about 36 (better than most of my competitors, worse than a few so it's a good headstart). The problem I would then have is a .com domain hosted on a Dutch server targeting UK people. Even if I was to set the webmaster tools location to UK it would not be ideal. Also, when it comes to launching the US site I would then be looking at using a .us domain which is far from ideal The other option I have is to use the .co.uk domain for the UK site but this is new and lack any decent moz score. Given this I am now pondering the following set up....using the .com domain on the Dutch server but putting the UK store in domain.com/UK and the future usa store in domain.com/usa. Would this be the best work around? I could then set the location of folders in the webmaster tools? Also, I plan on using geo redirecting on the domain so if a uk page happens to rank in the USA listings the user gets automatically redirected to the nearest matching product available in their country in the /us/ folder. Would this be easiest to work with on just one domain as it wouldn't technically be redirecting people to another site as per using two domains. Any thoughts would be good. Not even sure I have managed to explain it very clearly hehe
Technical SEO | | Grumpy_Carl0 -
Mobile site: robots.txt best practices
If there are canonical tags pointing to the web version of each mobile page, what should a robots.txt file for a mobile site have?
Technical SEO | | bonnierSEO0 -
Should I set up a disallow in the robots.txt for catalog search results?
When the crawl diagnostics came back for my site its showing around 3,000 pages of duplicate content. Almost all of them are of the catalog search results page. I also did a site search on Google and they have most of the results pages in their index too. I think I should just disallow the bots in the /catalogsearch/ sub folder, but I'm not sure if this will have any negative effect?
Technical SEO | | JordanJudson0