In all honesty people worry far too much about the duplicate content rule. Duplicate content is primarily for content sites and only penalized on a large scale.

In your case not only is the about page one that you likely aren't hoping to get ranked, but it's such a small factor that I doubt it will come into play.

Your inbound marketing strategy is likely to be much more than SEO traffic in fact I'd be willing to bed that a good 60%+ of your traffic is direct and/or social. In such a case as long as users are able to find the information they need on the site (and ultimately convert to bookings keeping the end-client happy and cash flowing) then I wouldn't worry about the "About Us" page.

Remember two things:

1. Put the customer experience first and everything else is gravy.

2. A good business (and good marketing) is a bigger picture than just SEO these days. Focus on what traffic origin makes sense for you and make sure they can access the information they need.

Here's what Matt Cutts had to say on the issue:
http://searchengineland.com/googles-matt-cutts-duplicate-content-wont-hurt-you-unless-it-is-spammy-167459

If you check out Rand's Intro to SEO slideshare (http://www.slideshare.net/randfish/introduction-to-seo-5003433) slide 46 and 47 talk about URL structure and specifically sub-domains.

As Rob said you do want to sub-folder structures and avoid sub-domains. Hopefully you are old enough to remember when websites like lycos.com were big and people could make their own websites. These were all hosted on subdomains like moz.tripod.lycos.com and because of this structure search engines needed to see subdomains as separate websites. For this reason they have separate grading, change the flow of link juice and can easily count as duplicate content.

Sub-domains are best utilized for information that is distinct enough. Like in the moz example Rands personal blog could theoretically sit at rand.moz.com as its a separate theme, different content, etc it would just loose out on the  flow of value.

Once again Rob is right about using 301 redirects to move your subdomains into folders.

Now moving on to the more specific nature of your question "Are fourth level sub-domains any worse than third level sub-domains" I am going to suggest that when asking such a question you've already lost a big chunk of the SEO/inbound marketing battle.

The question you are framing is "I know it isn't good - but is it any worse?" Well even if it's not any worse you already know that it's not great and you should be taking structural steps to build on a sites accessibility, user functionality and it's SEO. If you find yourself asking "Is X any worse?" "How bad is Y?" "Can I get away with Z?" then you should immediately stop pursuing that idea and try and find a different method.

In this case that method is sub-folders and a 301 migration, but remember the framing of your questions and your over all directional strategy need to change to really drive home your campaigns!

First, is this content dynamic? Depending on how the user progresses through these choices a search engine may not even see this information.

Second, as long as those aren't the pages you are trying to rank I can't see them actually having that big of an impact (if any) on your overall SEO. There is a difference between having pages that have the same content, and having pages that are duplicate content.

Rel canonical would be another way to save your skin, and making sure that the choices themselves are no-follow links (as long as there is some other method to get to the deeper pages).

Personally for the sake of SEO and user experience (which is far more important) I'd talk to the company about creating a dynamic selection wizard that could be a popup and once the user makes all their choices brings them to the right HTML page.

The HTML pages would be open, crawl-able, structured and sitemapped and the wizard would by a dynamic widget loaded in from a page that was robots disallowed. Helping both the human and spider experience - just my two cents!

Best of luck!

Hi Andrews,

While the difference between dashes and underscores use to be a big issue a few years back its something that seems to hold minimal merit now. The two can be used rather interchangeably without any major impact. This was phased out around the same time as exact-match-domain value was as far too many people were abusing the long-tail dash page method.

While I've never come across this exact problem before I can share with you one my mantras that applies here:

"If a system (browser, search engine, etc) needs to perform a data re-write, you aren't accessible enough."

Google loves accessibility. It always wants the user to be able to easily access information and it wants it's spiders to be able to easily index and categorize the information. When accessibility options such as javascript versioning or if a site is using flash or not have an impact then it would only logically follow that more obvious structural access issues do come into effect.

From a technology stand point I can tell you that "." is not traditionally used in the scope of a URL/file structure as it a reserved character and therefore your structure is being re-written to display those. Much like international domains like the chinese internationalized domain name extension .中国 (which is basically a visual re-encode of the unicode: xn--fiqs8s) For the sake of accessibility, proper structure formatting and system practicality you should avoid using non-standard characters such as the . in your url

Yup wildcard syntax is indeed still valid. However I can only confirm that the big 3 (Google, Yahoo and Bing) actively observe it. Other secondary search engines may not.

In your case you are probably looking for a syntax along the lines of:

User-agent: *
Disallow: /*.pdf$ This would set that any user agent should be blocked from any file name that ends in .pdf (a $ ties it to the end so pdf.txt would not be blocked in this case)

Keep an eye on how you block them. Missing a trailing slash could block a directory rather than a file, or not appending a strict symbol ($) could mean that phrases throughout a directory could be blocked rather than just a filename.

Also keep in mind if you are using URL re-writing this may play into how you need to block things; and you may also want to remember that disallowing access in a robot.txt does NOT prevent search engines from indexing the data, it is up to them if they honor the request. So if it is very important to block the file access from search engines then robots.txt may not be the way to do it.

This isn't so much an SEO question as it is a personal policy question.

Obviously declaring something as No-Follow allows you to retain the PR flow from your website. Yet this isn't always something that guest posters are happy with.

The other thing worth considering if your guest posters are only willing to write for you in exchange for a Do-Follow link is maybe they aren't the best guest posters. Not only will similar guest articles and do-follow links from them be popping up on a number of sites as this is their active strategy, but there end goal isn't user experience it's SEO.

To improve YOUR seo and ultimately your business you want a guest poster who cares more about improving their personal brand than their search engine ranking. This way you will find a user committed to quality and user experience, someone that search engines and people a like want to read, and someone who won't require you to give a do-follow link, because regardless of link status their brand grows.

Remember inbound marketing isn't all about SEO, you need to debate where the technical side of your business ends and the branding begins and how that effects your policy!

Due to the information we have this response is obviously going to be some educated speculation. You said 1.5 months ago that you changed the structure in how you present your language options to the user and I think this has a great deal to do with the index pages your seeing.

If you check out Rand's SEO slideshare (http://www.slideshare.net/randfish/introduction-to-seo-5003433) from slides 39-47 you'll see his discussion on the importance of site structure in the eyes of Google. While translated content may be all the same to the user, the search engines take the structure to mean different matters of intent.

For example, sub-domain information is often taken to be duplicated translate purpose only content. It's also often categorized as a separate site.

When you went from sub-domains to language directories you went from three separate sites to one site with flow-down accessible information. In Google's eyes you just expanded your website with new fresh and valuable information. While some of the indexed pages may drop off I think this structural change is the main reason you've had such a pick up in indexing and hopefully it plays well for your on your international SEO campaign!

Cheers,
Adam

If you check out Rand's Intro to SEO slideshare (http://www.slideshare.net/randfish/introduction-to-seo-5003433) slide 46 and 47 talk about URL structure and specifically sub-domains.

As Rob said you do want to sub-folder structures and avoid sub-domains. Hopefully you are old enough to remember when websites like lycos.com were big and people could make their own websites. These were all hosted on subdomains like moz.tripod.lycos.com and because of this structure search engines needed to see subdomains as separate websites. For this reason they have separate grading, change the flow of link juice and can easily count as duplicate content.

Sub-domains are best utilized for information that is distinct enough. Like in the moz example Rands personal blog could theoretically sit at rand.moz.com as its a separate theme, different content, etc it would just loose out on the  flow of value.

Once again Rob is right about using 301 redirects to move your subdomains into folders.

Now moving on to the more specific nature of your question "Are fourth level sub-domains any worse than third level sub-domains" I am going to suggest that when asking such a question you've already lost a big chunk of the SEO/inbound marketing battle.

The question you are framing is "I know it isn't good - but is it any worse?" Well even if it's not any worse you already know that it's not great and you should be taking structural steps to build on a sites accessibility, user functionality and it's SEO. If you find yourself asking "Is X any worse?" "How bad is Y?" "Can I get away with Z?" then you should immediately stop pursuing that idea and try and find a different method.

In this case that method is sub-folders and a 301 migration, but remember the framing of your questions and your over all directional strategy need to change to really drive home your campaigns!

Due to the information we have this response is obviously going to be some educated speculation. You said 1.5 months ago that you changed the structure in how you present your language options to the user and I think this has a great deal to do with the index pages your seeing.

If you check out Rand's SEO slideshare (http://www.slideshare.net/randfish/introduction-to-seo-5003433) from slides 39-47 you'll see his discussion on the importance of site structure in the eyes of Google. While translated content may be all the same to the user, the search engines take the structure to mean different matters of intent.

For example, sub-domain information is often taken to be duplicated translate purpose only content. It's also often categorized as a separate site.

When you went from sub-domains to language directories you went from three separate sites to one site with flow-down accessible information. In Google's eyes you just expanded your website with new fresh and valuable information. While some of the indexed pages may drop off I think this structural change is the main reason you've had such a pick up in indexing and hopefully it plays well for your on your international SEO campaign!

Cheers,
Adam

This isn't so much an SEO question as it is a personal policy question.

Obviously declaring something as No-Follow allows you to retain the PR flow from your website. Yet this isn't always something that guest posters are happy with.

The other thing worth considering if your guest posters are only willing to write for you in exchange for a Do-Follow link is maybe they aren't the best guest posters. Not only will similar guest articles and do-follow links from them be popping up on a number of sites as this is their active strategy, but there end goal isn't user experience it's SEO.

To improve YOUR seo and ultimately your business you want a guest poster who cares more about improving their personal brand than their search engine ranking. This way you will find a user committed to quality and user experience, someone that search engines and people a like want to read, and someone who won't require you to give a do-follow link, because regardless of link status their brand grows.

Remember inbound marketing isn't all about SEO, you need to debate where the technical side of your business ends and the branding begins and how that effects your policy!

Yup wildcard syntax is indeed still valid. However I can only confirm that the big 3 (Google, Yahoo and Bing) actively observe it. Other secondary search engines may not.

In your case you are probably looking for a syntax along the lines of:

User-agent: *
Disallow: /*.pdf$ This would set that any user agent should be blocked from any file name that ends in .pdf (a $ ties it to the end so pdf.txt would not be blocked in this case)

Keep an eye on how you block them. Missing a trailing slash could block a directory rather than a file, or not appending a strict symbol ($) could mean that phrases throughout a directory could be blocked rather than just a filename.

Also keep in mind if you are using URL re-writing this may play into how you need to block things; and you may also want to remember that disallowing access in a robot.txt does NOT prevent search engines from indexing the data, it is up to them if they honor the request. So if it is very important to block the file access from search engines then robots.txt may not be the way to do it.

While I've never come across this exact problem before I can share with you one my mantras that applies here:

"If a system (browser, search engine, etc) needs to perform a data re-write, you aren't accessible enough."

Google loves accessibility. It always wants the user to be able to easily access information and it wants it's spiders to be able to easily index and categorize the information. When accessibility options such as javascript versioning or if a site is using flash or not have an impact then it would only logically follow that more obvious structural access issues do come into effect.

From a technology stand point I can tell you that "." is not traditionally used in the scope of a URL/file structure as it a reserved character and therefore your structure is being re-written to display those. Much like international domains like the chinese internationalized domain name extension .中国 (which is basically a visual re-encode of the unicode: xn--fiqs8s) For the sake of accessibility, proper structure formatting and system practicality you should avoid using non-standard characters such as the . in your url

Hi Andrews,

While the difference between dashes and underscores use to be a big issue a few years back its something that seems to hold minimal merit now. The two can be used rather interchangeably without any major impact. This was phased out around the same time as exact-match-domain value was as far too many people were abusing the long-tail dash page method.

First, is this content dynamic? Depending on how the user progresses through these choices a search engine may not even see this information.

Second, as long as those aren't the pages you are trying to rank I can't see them actually having that big of an impact (if any) on your overall SEO. There is a difference between having pages that have the same content, and having pages that are duplicate content.

Rel canonical would be another way to save your skin, and making sure that the choices themselves are no-follow links (as long as there is some other method to get to the deeper pages).

Personally for the sake of SEO and user experience (which is far more important) I'd talk to the company about creating a dynamic selection wizard that could be a popup and once the user makes all their choices brings them to the right HTML page.

The HTML pages would be open, crawl-able, structured and sitemapped and the wizard would by a dynamic widget loaded in from a page that was robots disallowed. Helping both the human and spider experience - just my two cents!

Best of luck!

In all honesty people worry far too much about the duplicate content rule. Duplicate content is primarily for content sites and only penalized on a large scale.

In your case not only is the about page one that you likely aren't hoping to get ranked, but it's such a small factor that I doubt it will come into play.

Your inbound marketing strategy is likely to be much more than SEO traffic in fact I'd be willing to bed that a good 60%+ of your traffic is direct and/or social. In such a case as long as users are able to find the information they need on the site (and ultimately convert to bookings keeping the end-client happy and cash flowing) then I wouldn't worry about the "About Us" page.

Remember two things:

1. Put the customer experience first and everything else is gravy.

2. A good business (and good marketing) is a bigger picture than just SEO these days. Focus on what traffic origin makes sense for you and make sure they can access the information they need.

Here's what Matt Cutts had to say on the issue:
http://searchengineland.com/googles-matt-cutts-duplicate-content-wont-hurt-you-unless-it-is-spammy-167459