Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Do I need a separate robots.txt file for my shop subdomain?
-
Hello Mozzers!
Apologies if this question has been asked before, but I couldn't find an answer so here goes...
Currently I have one robots.txt file hosted at https://www.mysitename.org.uk/robots.txt
We host our shop on a separate subdomain https://shop.mysitename.org.uk
Do I need a separate robots.txt file for my subdomain? (Some Google searches are telling me yes and some no and I've become awfully confused!
-
Thank you. I want to disallow specific URLs on the subdomain and add the shop sitemap in the robots.txt file. So I'll go ahead and create another!
-
You go be fine without one. You only need one if you want to manage that subdmain: add specific xml sitemaps links in robots.txt, cut access to specific folders for that subdomain.
if you don't need any of that - just move forward without one.
-
Currently we just have: User-agent: *
I'm in the process of optimising.
-
It depends what currently is in your robots.txt. Usually it would be useful to have another one for your subdomain.
-
Yes, I would have a seperate robots.txt files.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When should a variant be a variant and when should it be a separate product from an SEO POV?
Hi all, We are looking at changing our current e-commerce store to a new platform and in doing so thinking of making some changes to how we list products in sub-categories. We have seen related questions asking about splitting a single product into multiple products to rank for different terms, but we are wondering about combining multiple products into a single product page? The examples we have seen have been about fashion items with variants of colour and size. However, the products we sell have variances that change the appearance, dimensions and technical specification, so we would like to ask the MOZ community if combining products with these variances would still be deemed good practice? We sell wood burning stoves and a good example of a product that we are considering combining is the Scan 85 stove, which is available in eight different configurations: 85-1, 85-2, 85-3 etc. Scan themselves refer to each version as a separate product and they are bought, stocked and sold as separate products. Wood burning stoves like this typically have a firebox in the centre and then design options that can change the top, side, base, door, colour and fuel. In this example, the firebox is the Scan 85 and the variation is the last number, each of which corresponds to a different design option changing both the appearance and dimensions (see attached image). We have them listed as eight different products on our current site, one for each version. Primarily because each option has its own name (albeit 1-digit difference) which when we created the pages we thought that more pages would present us with more ranking opportunity. However, we have since learnt that because these eight pages are all so similar and it is difficult to write unique content about each product (with the 85-1 and 85-2 the only difference between the models are the black trim on the 85-1 and the silver trim on 85-2). Especially as when talking about the firebox itself, how well the fire burns, how controllable it is etc, will be the same for all versions. Likewise, earning backlinks to eight separate pages is also very difficult. Exploring this lead, us to the question, when is a variant a variant and when is it a separate product? Are there hard and fast rules for what defines variants and products? Or does it simply vary from industry to industry product to product, and if so should we be looking at it from a UX or SEO POV, when making that decision? Our hope is that if we combine these eight products into a single high-quality page, it will present us with a greater ranking opportunity for that one page over eight individual pages. We also hope that in doing so will allow us to create a more intuitive UX on a single page with a unique description, more reviews focused on one page and an explanation of the options available, all of which should lead to more conversions. Finally, by creating a better UX and unique detailed description we hope that there is a higher chance of us earning product level backlinks then we do with eight lower quality pages. One of the issues in creating a single product page for all the variants is the sub-category/results pages, as we would be removing eight simple products and replacing them with one complex product. We have questions over how this would work from a filter/facet level whereby when you apply a filter there is an expectation that the image shown will match the criteria, so if we filter for stoves with a silver trim for example, there is an expectation to only see stoves that have a silver trim in the results. When you have separate product pages you have separate listings which makes this easier to only bring back the models matching the criteria. However, when you have a single page this is more complex as you will need a default image for non-filtered results and then the ability to assign an image to lots of different attributes so that the correct image is always shown that matches the criteria selected. All of which we have been assured is do-able but adds an extra level of complexity to the process from an admin side. The alternative to doing this would be to create eight simple/child products and link them to one configurable/parent product. We could them list the simple products into the results pages and have them all linking back to the main configurable product which could load with the options of the simple product that was selected. From an SEO POV this brings in some more work, redirecting each page to the parent, but ultimately this could provide a better UX and might be the better solution. Has anyone got any experience in doing either of these options before? Both options above with affect the number of products we have available, so does the number of products in a sub-category effect the ability for that category page to rank? We currently have around 500 products in our wood burning stoves category, with perhaps an additional 300 to add. If we go down the combining into a single product page route this will reduce the number of products by around a third. If we keep all the simple/child products, then this will stay around the same. So, have we missed something obvious? Is there a glaring issue that we have overlooked from an SEO point of view as well as from the customer experience? We would appreciate your thoughts on this. Thanks, Reece scan85-1.jpg
Technical SEO | | fireproductsuk0 -
Robots.txt & meta noindex--site still shows up on Google Search
I have set up my robots.txt like this: User-agent: *
Technical SEO | | RoxBrock
Disallow: / and I have this meta tag in my on a Wordpress site, set up with SEO Yoast name="robots" content="noindex,follow"/> I did "Fetch as Google" on my Google Search Console My website is still showing up in the search results and it says this: "A description for this result is not available because of this site's robots.txt" This site has not shown up for years and now it is ranking above my site that I want to rank for this keyword. How do I get Google to ignore this site? This seems really weird and I'm confused how a site with little content, that has not been updated for years can rank higher than a site that is constantly updated and improved.1 -
Does Google index internal anchors as separate pages?
Hi, Back in September, I added a function that sets an anchor on each subheading (h[2-6]) and creates a Table of content that links to each of those anchors. These anchors did show up in the SERPs as JumpTo Links. Fine. Back then I also changed the canonicals to a slightly different structur and meanwhile there was some massive increase in the number of indexed pages - WAY over the top - which has since been fixed by removing (410) a complete section of the site. However ... there are still ~34.000 pages indexed to what really are more like 4.000 plus (all properly canonicalised). Naturally I am wondering, what google thinks it is indexing. The number is just way of and quite inexplainable. So I was wondering: Does Google save JumpTo links as unique pages? Also, does anybody know any method of actually getting all the pages in the google index? (Not actually existing sites via Screaming Frog etc, but actual pages in the index - all methods I found sadly do not work.) Finally: Does somebody have any other explanation for the incongruency in indexed vs. actual pages? Thanks for your replies! Nico
Technical SEO | | netzkern_AG0 -
Does an subdomain hosted offsite provide SEO value
We have a job board hosted through an applicant processing system which we've setup as a subdomain (jobs.ourcompany.com), most of the assets are hosted on our primary domain (ourcompany.com). My question is does having it hosted offsite provide any value? Do we get credit for that content being shared and distributed on the web or does the applicant processing system? As I see it the options are (correct me if I'm wrong): Host the job listings on our primary domain (ourcompany.com/jobs) and have it point to the application on the subdomain. Advertise the job listings pointing to the primary domain on the paid sites. The free job listing sites will automatically point to the sub-domain because the applicant processing system automatically submits them. Host the job listings entirely on the sub-domain applicant tracking system and link to it from our primary site navigation. Advertise the job listings to the sub-domain so that both free and paid point to the same place. Obviously the second one would be much easier just not sure on the technical side of our website getting credit by search engines as the one who has produced the content.
Technical SEO | | r1200gsa0 -
Empty Meta Robots Directive - Harmful?
Hi, We had a coding update and a side-effect of that was that our directive was emptied, in other words it now reads as: on all of the site. I've since noticed that Google's cache date on all of the pages - at least, the ones I tested - have a Cached date of no later than 17 December '12 - that's the Monday after the directive was removed on mass. So, A, does anyone have solid evidence of an empty directive causing problems? Past experience, Matt Cutts, Fishkin quote, etc. And then B - It seems fairly well correlated but, does my entire site's homogenous Cached date point to this tag removal? Or is it fairly normal to have a particular cache date across a large site (we're a large ecommerce site). Our site: http://www.zando.co.za/ I'm having the directive reinstated as soon as Dev permitting. And then, for extra credit, is there a way with Google's API, or perhaps some other tool, to run an arbitrary list and retrieve Cached dates? I'd want to do this for diagnosis purposes and preferably in a way that OK with Google. I'd avoid CURLing for the cached URL and scraping out that dates with BASH, or any such kind of thing. Cheers,
Technical SEO | | RocketZando0 -
Oh no googlebot can not access my robots.txt file
I just receive a n error message from google webmaster Wonder it was something to do with Yoast plugin. Could somebody help me with troubleshooting this? Here's original message Over the last 24 hours, Googlebot encountered 189 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%. Recommended action If the site error rate is 100%: Using a web browser, attempt to access http://www.soobumimphotography.com//robots.txt. If you are able to access it from your browser, then your site may be configured to deny access to googlebot. Check the configuration of your firewall and site to ensure that you are not denying access to googlebot. If your robots.txt is a static page, verify that your web service has proper permissions to access the file. If your robots.txt is dynamically generated, verify that the scripts that generate the robots.txt are properly configured and have permission to run. Check the logs for your website to see if your scripts are failing, and if so attempt to diagnose the cause of the failure. If the site error rate is less than 100%: Using Webmaster Tools, find a day with a high error rate and examine the logs for your web server for that day. Look for errors accessing robots.txt in the logs for that day and fix the causes of those errors. The most likely explanation is that your site is overloaded. Contact your hosting provider and discuss reconfiguring your web server or adding more resources to your website. After you think you've fixed the problem, use Fetch as Google to fetch http://www.soobumimphotography.com//robots.txt to verify that Googlebot can properly access your site.
Technical SEO | | BistosAmerica0 -
Duplicate content problem from an index.php file
Hi One of my sites is flagging a duplicate content problem which is affecting the search rankings. The duplicate problem is caused by http://www.mydomain.com/index.php which has a page rank of 26 How can I sort the duplicate content problem, as the main page should just be http://www.mydomain.com which has a page rank of 42 and is the stronger page with stronger links etc Many Thanks
Technical SEO | | ocelot0 -
What are your best tips for SEO on a shopping cart?
So, I am working on a shopping cart platform (X-Cart) and so far don't like it. Also, the web designer is not someone I've worked with before and he is understandably conservative about access--which limits what I can and cannot do from the back end. One of the things I like to do is include text for the search engines. However, based on conversion, etc., I think the product images on a landing page (main brand info with specific products that show up) should show up first to move toward conversion first. I am thinking of adding the text below the product images on the brand pages so the viewer sees the products first while still keeping the content seo. My practice is to use between 300-350 words minimum on a page. Just wondering what best practices you have for a shopping cart. Care to share? Any tips or hints? Thoughts on what I might do that would be most effective? As always, thanks in advance for your sage advice!
Technical SEO | | TheARKlady0