How to resolve Duplicate Page Content issue for root domain & index.html?

[ContentWriterMicky]

SEOMoz returns a Duplicate Page Content error for a website's index page, with both domain.com and domain.com/index.html isted seperately. We had a rewrite in the htacess file, but for some reason this has not had an impact and we have since removed it. What's the best way (in an HTML website) to ensure all index.html links are automatically redirected to the root domain and these aren't seen as two separate pages?

[willyg]

great code Josh...but , after i saved it on .htaccess , a "?" appeared on the link..

http://www.domain.com/?/example/file.html

Is this ok ? pls advice/

Thank you,

[SEMPassion]

You touched on a good point here "We set  up our site to utilize a index redirect for all of our sub directories as well, so with this method you simply name your sub directories to match the url path that you desire. Each sub directory has it's own index which you redirect with a variation of the above code. By doing this you can have nice clean url paths like http://www.semclix.com/design/ecommerce/ - and mitigate the duplicate content issue. We hope that this helps."

Too often I see sites where they get the home page right but miss the re-write on the directories.

[SEMCLIX]

Here's the .htaccess rewrite command that you can use for the index.html redirect -

Options +FollowSymlinks                                                                                                   RewriteEngine on

Index Rewrite                                                                                                                   RewriteRule ^index.(htm|html|php) http://www.amarasoftware.com/ [R=301,L]              RewriteRule ^(.*)/index.(htm|html|php) http://www.amarasoftware.com/$1/ [R=301,L]

---

We set  up our site to utilize a index redirect for all of our sub directories as well, so with this method you simply name your sub directories to match the url path that you desire. Each sub directory has it's own index which you redirect with a variation of the above code. By doing this you can have nice clean url paths like http://www.semclix.com/design/ecommerce/ - and mitigate the duplicate content issue. We hope that this helps.

[SteveOllington]

I'd check it with some other software too... i.e. Raven Tools free trial or something, that will tell you if there's canonicalization problems... of course I'm not advocating Raven Tools over SEOmoz tools (I'm a member here and not there for good reasons), I just think best to try a few different tests before deciding if it's a problem. There might just be an issue with the SEOmoz campaign tool for the moment, which I'm sure they'll fix as soon as they realise.

Hey, aren't you the tutor I had in my SEC usability course?

[BedeFahey]

Unfortunately I can't speak for how SEOmoz handles rewrites like this if it's already crawled the page.

The rewrite rule you're using looks like it's only rewriting the www portion of the URL, not index.html. So alone it wouldn't do anything to solve dupe content issues. (someone please correct me if I'm misreading the rewrite rule)

Here's a link to what I used to write a redirect for index.html on another site.

http://www.webmasterworld.com/forum92/6375.htm

I think it is a fairly safe assumption to make that SEOmoz is smart enough to realize if you're got a redirect in there (providing that its working). I'd still recommend taking a look to see if Google has cached or indexed an index.html version, though.

Edit: my personal, highly technical, acid-test for an index.html redirect is just going there and manually entering the url with index.html on the end, rather than waiting for a recrawl to see if you're heading in the right direction.

[ContentWriterMicky]

RewriteEngine on RewriteCond %{HTTP_HOST} ^([a-z.]+)?amarasoftware.com$ [NC] RewriteCond %{HTTP_HOST} !^www. [NC] RewriteRule .? http://www.%1amarasoftware.com%{REQUEST_URI} [R=301,L] Is what I use. In Seomoz this leads to www.amarasoftware.com and index.html so 2 different URL's, both with different incoming links, and a different authority, which has an impact on my ranking if correct. in SEomoz this a returns a duplicate title and meta tags errors. If SEOmoz finds 2 pages instead of one I may assume that Google agrees with this.

[BedeFahey]

As you did, I'd normally handle this with a 301 from index.html to the root domain. When you say that it's "not had an impact" do you mean that the SEOmoz dashboard continues to show an error after it re-crawls, or that the search engines are not picking up the redirect?

SEOmoz dashboard does a great job, but I'd check to see how the search engines are actually indexing yourdomain.com/index.html vs. yourdomain.com also. If the search engines are indexing it as you want them to, then I'd be inclined to ignore the dashboard error.

I apologize if this is a stupid question, but I assume you manually checked that the redirect worked?

[RyanKent]

You wish to canonicalize the pages. That is the SEO word which describes exactly what you are trying to achieve.

domain.com

domian.com/

www.domain.com

www.domain.com/

domain.com/index.html

Above are 5 URLs which can possibly lead to the exact same page. If you add the following HTML in the code then the pages will be canonicalized.