How to handle a blog subdomain on the main sitemap and robots file?
-
Hi,
I have some confusion about how our blog subdomain is handled in our sitemap. We have our main website, example.com, and our blog, blog.example.com.
-
Should we list the blog subdomain URL in our main sitemap? In other words, is listing a subdomain allowed in the root sitemap?
-
What does the final structure look like in terms of the sitemap and robots file? Specifically:
would I include a link to our blog subdomain (blog.example.com)?
would I include a link to BOTH our main sitemap and blog sitemap?
would I include a link to our main website URL (even though it's not a subdomain)?
does a subdomain need its own robots file?
I'm a technical SEO and understand the mechanics of much of on-page SEO.... but for some reason I never found an answer to this specific question and I am wondering how the pros do it. I appreciate your help with this.
-
-
It's my understanding that you treat each subdomain as a unique site. So each subdomain should have its own unique XML sitemap and robots.txt file, as well as submitted separately to Google Webmaster Tools. But to answer your inter-linking question, I would avoid including the other domain URLs in those files (XML and Robots). Only include the URLs for that particular domain and/or subdomain. With that being said however, I would inter-link them on the actual site somewhere. Maybe in the HTML sitemap, navigation, footer, or even naturally throughout your body content where appropriate as well.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt for Facet Results
Hi Does anyone know how to properly add facets URL's to Robots txt? E.g. of our facets URL - http://www.key.co.uk/en/key/platform-trolleys-trucks#facet:-10028265807368&productBeginIndex:0&orderBy:5&pageView:list& Everything after the # will need to be blocked on all pages with a facet. Thank you
Intermediate & Advanced SEO | | BeckyKey0 -
Subdomain SEO question (php script on domain + wordpress on subdomain)
Hi Moz fellows, I am doing my first website which is entirely .php scripted. But I would like to have a wordpress blog to create content and blog posts, while the .php side of the website is more for sales pages and user generated listings.The only way to do this is to install wordpress on a subdomain "blog.website.com" QUESTION: If all my keywords targeted content is on the subdomain's Wordpress blog, but all my guest blogging efforts link to my main website, which one will rank? The subdomain or the domain? I need the domain to rank well as it is a Fiverr-like script, so if tons of people land on my "blog.website.com" subdomain, they will not convert into users... Let me know if you have experience with such a scenario, and thank you all in advance for your help! -Marc
Intermediate & Advanced SEO | | marcandre0 -
Url structure of a blog
We are trying to work out what the best structure for our blog is as we want each page to rank as highly as possible, we were looking at a flat structure similar to http://www.hongkiat.com/blog/ where every posts is after the blog/ but not in category's although the viewers can look in different category's from the top buttons on the page- photoshop - icons etc or we where going to go for the structured way- blog/photoshop/blog-post.html the only problem is that we will end up 4 deep at least with this and at least 80 characters in the url. any help would be appreciated. Thanks Shaun
Intermediate & Advanced SEO | | BobAnderson0 -
Indexing/Sitemap - I must be wrong
Hi All, I would guess that a great number of us new to SEO (or not) share some simple beliefs in relation to Google indexing and Sitemaps, and as such get confused by what Web master tools shows us. It would be great if somone with experience/knowledge could clear this up for once and all 🙂 Common beliefs: Google will crawl your site from the top down, following each link and recursively repeating the process until it bottoms out/becomes cyclic. A Sitemap can be provided that outlines the definitive structure of the site, and is especially useful for links that may not be easily discovered via crawling. In Google’s webmaster tools in the sitemap section the number of pages indexed shows the number of pages in your sitemap that Google considers to be worthwhile indexing. If you place a rel="canonical" tag on every page pointing to the definitive version you will avoid duplicate content and aid Google in its indexing endeavour. These preconceptions seem fair, but must be flawed. Our site has 1,417 pages as listed in our Sitemap. Google’s tools tell us there are no issues with this sitemap but a mere 44 are indexed! We submit 2,716 images (because we create all our own images for products) and a disappointing zero are indexed. Under Health->Index status in WM tools, we apparently have 4,169 pages indexed. I tend to assume these are old pages that now yield a 404 if they are visited. It could be that Google’s Indexed quotient of 44 could mean “Pages indexed by virtue of your sitemap, i.e. we didn’t find them by crawling – so thanks for that”, but despite trawling through Google’s help, I don’t really get that feeling. This is basic stuff, but I suspect a great number of us struggle to understand the disparity between our expectations and what WM Tools yields, and we go on to either ignore an important problem, or waste time on non-issues. Can anyone shine a light on this for once and all? If you are interested, our map looks like this : http://www.1010direct.com/Sitemap.xml Many thanks Paul
Intermediate & Advanced SEO | | fretts0 -
Effect duration of robots.txt file.
in my web site there is demo site in that also, index in Google but no need it now.so i have created robots file and upload to server yesterday.in the demo folder there are some html files,and i wanna remove all these in demo file from Google.but still in web master tools it showing User-agent: *
Intermediate & Advanced SEO | | innofidelity
Disallow: /demo/ How long this will take to remove from Google ? And are there any alternative way doing that ?0 -
Should I redirect all my subdomains to a single unique subdomain to eliminate duplicate content?
Hi there! I've been working on http://duproprio.com for a couple of years now. In the early stages of the website, we've put into place a subdomain wildcard, that allowed us to create urls like this on the fly : http://{some-city}.duproprio.com This brought us instantly a lot of success in terms of traffic due to the cities being great search keywords. But now, business has grown, and as we all know, duplicate content is the devil so I've been playing with the idea of killing (redirecting) all those urls to their equivalent on the root domain. http://some-city.duproprio.com/some-listing-1234 would redirect to equivalent page at : http://duproprio.com/some-listing-1234 Even if my redirections are 301 permanent, there will be some juice lost for each link redirected that are actually pointing to my old subdomains This would also imply to redirect http://www.duproprio.com to http://duproprio.com. Which is probably the part I'm most anxious about since the incoming links are almost 50/50 between those 2 subdomains... Bringing everything back into a single subdomain is the thing to do in order to get all my seo juice together, this part is obvious... But what can I do to make sure that I don't end up actually losing traffic instead of gaining authority? Can you help me get the confidence I need to make this "move" without risking to lose tons of traffic? Thanks a big lot!
Intermediate & Advanced SEO | | DuProprio.com0 -
Duplicating an article I wrote on an external blog
Hi, I wrote a blog article on another site. I would like to add the article to my site as well and would like to know the best way to do it. If I duplicate the article that I wrote would I then risk getting a penalty for duplicate content? If so, then what is the best way for me to include the article on my site for the benefit of my readers, but not lead to the duplicate content problem? Would it be better to use a canonical tag? Or to noindex the page? If I use the canonical tag, am I helping to make the article on the external blog stronger? Where is I use the noindex tag I am not helping my site nor that article I think, is that right? Last question, if I offer the copy of the article on my site and use the canonical or noindex tag then my site does not receive any direct benefit from the article for SEO. In other words the article wont appear in the search index with a link to my site. What about the comments that people write on the article on my site? That is unique content which may have great questions or points. I want to ensure those can be indexed properly. If I noindex the page I lose out. If I canonicalize (is that a word?) the page then I don't know if will send search results based on those comments to the external blog where that information (the comments from my site) does not exist. Thank you for any help to better understand this part of seo.
Intermediate & Advanced SEO | | NikkiGaul0 -
How to get subdomains to rank well?
Hi All, I am setting up a new site and I want to make use of subdomains to target multiple countries as follows: uk.mydomain.com us.mydomain.com australia.mydomain.com etc. Now i know what you're all going to say, why not use folders as they are more effective. Well I did think of this but decided against it because I would like to make the best of a low competition industry. I want to push my competitors as far down in the SE's as possible and i plan to do this by targeting generic non locational search terms with both sites so I can hog the top 4 spots.as follows: www.mydomain.com www.mydomain.com/keyterm uk.mydomain.com uk.mydomain.com/keyterm-in-the-UK Whats steps can I take to ensure rank passes to my subdomains? Is it better to start the site with folders like www.mydomain.com/us/keyterm and then 301 them to subdomains at a later stage or should i start with the subdomains?
Intermediate & Advanced SEO | | Mulith1