Sub Domains and Robot.txt files...
-
This is going to seem like a stupid question, and perhaps it is but I am pulling out what little hair I have left.
I have a sub level domain on which a website sits. The Main domain has a robots.txt file that disallows all robots. It has been two weeks, I submitted the sitemap through webmaster tools and still, Google has not indexed the sub domain website. My question is, could the robots.txt file on the main domain be affecting the crawlability of the website on the sub domain? I wouldn't have thought so but I can find nothing else.
Thanks in advance.
-
Thank you, Mr. Young. I believed this to be the case (that it wasn't the robots.txt file) but I could think of nothing else. I have since been indexed.
-
The way that Google finds robots.txt files is by taking your URL, and adding /robots.txt to it. So a good way to see if the robots.txt file is affecting your subdomain is to go to subdomain.domain.com/robots.txt. If the file exists, then it is affecting your subdomain. If it doesn't, then it's only active on your main domain.
Getting indexed is function of having unique content and pagerank, so make sure your subdomain has unique content and links if you're having trouble getting it indexed. Submitting a sitemap is no guarantee that Google will index your site.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website of Products by a Brand on a Sub-Domain or a Sub-Directory?
Hi, We have a brand website under which we plan to develop various apps. We think that putting each website on its own sub-domain would be a good option. E.g. Main website – abc.com; app1 website – app1.abc.com; app2 website – app2.abc.com etc. Would it be the right way to do it or should we put the apps in a sub-directory. E.g. abc.com/app1; abc.com/app2 etc. These apps would either be totally unrelated or may be in future we may provide something so as to enable their integration. Thanks
Technical SEO | | IM_Learner0 -
Should I block Map pages with robots.txt?
Hello, I have a website that was started in 1999. On the website I have map pages for each of the offices listed on my site, for which there are about 120. Each of the 120 maps is in a whole separate html page. There is no content in the page other than the map. I know all of the offices love having the map pages so I don't want to remove the pages. So, my question is would these pages with no real content be hurting the rankings of the other pages on our site? Therefore, should I block the pages with my robots.txt? Would I also have to remove these pages (in webmaster tools?) from Google for blocking by robots.txt to really work? I appreciate your feedback, thanks!
Technical SEO | | imaginex0 -
Robots.txt Question
In the past, I had blocked a section of my site (i.e. domain.com/store/) by placing the following in my robots.txt file: "Disallow: /store/" Now, I would like the store to be indexed and included in the search results. I have removed the "Disallow: /store/" from the robots.txt file, but approximately one week later a Google search for the URL produces the following meta description in the search results: "A description for this result is not available because of this site's robots.txt – learn more" Is there anything else I need to do to speed up the process of getting this section of the site indexed?
Technical SEO | | davidangotti0 -
Campaigns Domain and Subdomain... ?
I made two separate campaigns before I understood the meaning of "subdomain". I make one campaign for my www.com and another for my .com. I now realize I should have made the .com the domain and the www. the subdomain in the same campaign. Is there a way to edit this? Thanks!
Technical SEO | | musicforkids0 -
Robots.txt
Hi there, My question relates to the robots.txt file. This statement: /*/trackback Would this block domain.com/trackback and domain.com/fred/trackback ? Peter
Technical SEO | | PeterM220 -
Need Help With Robots.txt on Magento eCommerce Site
Hello, I am having difficulty getting my robots.txt file to be configured properly. I am getting error emails from Google products stating they can't view our products because they are being blocked, and this past week, in my SEO dashboard, the URL's receiving search traffic dropped by almost 40%. Is there anyone that can offer assistance on a good template robots.txt file I can use for a Magento eCommerce website? The one I am currently using was found at this site here: e-commercewebdesign.co.uk/blog/magento-seo/magento-robots-txt-seo.php - However, I am getting problems from Google now because of it. I searched and found this thread here: http://www.magentocommerce.com/wiki/multi-store_set_up/multiple_website_setup_with_different_document_roots#the_root_folder_robots.txt_file - But I felt like maybe I should get some additional help on properly configuring a robots for a Magento site. Thanks in advance for any help. Please, let me know if you need more info to provide assistance.
Technical SEO | | JerDoggMckoy0 -
Question about domain redirects
One of my clients has an odd domain redirect situation. See if you can get your head round this: Domain A is set-up as a domain alias of Domain B Entering domain A or domain B takes you to default.asp on domain B. The default.asp includes VB script to check the HTTP_HOST variable. It checks whether the main doman name for domain A is present in the HTTP_HOST and if so redirects it to domain A/sub-folder/index.htm. If not present it redirects to domain B/index.htm. In both cases the redirect uses a response.Redirect clause. I think what is trying to be achieved is to redirect requests to Domain A to a sub-folder of Domain B. It works but seems extremely convoluted. Can anyone see problems with this set-up? Will link juice be lost along the redirect paths?
Technical SEO | | bjalc20110