Duplicate Content for index.html
-
In the Crawl Diagnostics Summary, it says that I have two pages with duplicate content which are:
I read in a Dream Weaver tutorial that you should name your home page "index.html" and then you can let www.mywebsite.com automatically direct the user to index.html. Is this a bug in SEOMoz's crawler or is it a real problem with my site?
Thank you,
Dan
-
The code should definitely go into the websites root directory's .htaccess, however .htaccess can be weird, a few days ago I ran into a similar issue with a client's website, and I was able to remedy the issue with a variation of the code.
index Redirect RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)index.(php|html|htm|asp)\ HTTP/ RewriteRule ^(([^/]+/))index.(php|html|htm|asp)$ http://yoursite.com/$1 [R=301,L]
If you give me the URL for the site I will take a look at it and let you know what would be feasible.
-
Hi Daniel, can you share with us the URL of your site? We can take a look at it and give you a more precise answer that way. Thanks!
-
I eventually figured out that your method was a 301 redirect and I definitely broke my site trying to use the code you posted. .. haha. Its ok though. I just removed the code and it went back to normal. At first, I was editing the .htaccess file in the public_html folder which wasnt working. Then I tried the root folder for the site (I created the .htaccess file since it did not exist.) Neither of those worked. (I am using Bluehost so I do not think that I have root access and I am not sure if it is a Linux server or not.)
If there is an easy way to explain what I am doing wrong, please do so. Otherwise, I will use canonical.
Thanks for everything!
-
@Dan
Thanks for your reply. It seems like there are lots of different ways to solve this problem. I just watched this video on Matt Cutt's blog where he discusses his preference for 301 redirects over rel canonical tag.
Where would you say your solution fits in?
sorry about the delay of this response, i didn't realize the that you were asking me a question right away. When placing the code I provided in my previous answer this will cause a 301 perminant redirect to the original URL. That's actually what the
[R=301,L]
portion of the code is stating (R) redirect (301) status is referring to. After reviewing the Matt Cutts video, I realize that I should have asked you if you were operating on a Linux server that you had root access to. We actually utilize both redirects and canonical tags since it was recommended by the on-page optimization reports. Heck Google uses them, I would assume because it's easier for the user to be referred to a single page URL. Obviously though if you don't have server header access, and are not familiar with .htaccess (you can accidentally break your site) then the canonical solution is appropriate
-
Josh,
Thanks for your reply. It seems like there are lots of different ways to solve this problem. I just watched this video on Matt Cutt's blog where he discusses his preference for 301 redirects over rel canonical tag.
Where would you say your solution fits in?
Thanks,
Dan -
use the link rel tag for all my homepages for the http://www.yoursite.com
-
Odd enough I just recently answered this question. The SEOmoz crawler is correct, because without a redirect you will be able to access both versions of the page in your browser.
To resolve this issue simply rewrite the index.html to the root url by placing the following code into your .htaccess file into your root directory.
Options +FollowSymlinks RewriteEngine on
Index Rewrite RewriteRule ^index.(htm|html|php) http://www.yoursite.com/ [R=301,L] RewriteRule ^(.*)/index.(htm|html|php) http://www.yoursite.com/$1/ [R=301,L]
You can also do the same with the index file in any subdirectories that you might create, by simply placing a .htaccess into those sub directories and using variations of the above code. This is how you create nice tight URLs without the duplicate content issue that look like - http://www.semclix.com/design/business/
-
It is a problem which you need to fix. You need to canonicalize your pages.
Those are all various URLs which most likely lead to the same web page. I say "most likely" because these URLs can actually lead to different pages.
You need to tell crawlers and search engines how you organize your site. There are several ways to achieve canonicalization. The method I prefer is to add the following line of code to each page:
The URL provided should be the preferred URL for your page.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When rel canonical tag used, which page does Google considers for ranking and indexing? A/B test scenario!
Hi Moz community, We have redesigned our website and launched for A/B testing using canonical tags from old website to new website pages, so there will be no duplicate content issues and new website will be shown to the half of the website visitors successfully to calculate the metrics. However I wonder how actually Google considers it? Which pages Google will crawl and index to consider for ranking? Please share your views on this for better optimisation. Thanks
Web Design | | vtmoz0 -
Multiple sites using same text - how to avoid Google duplicate content penalty?
Hi Mozers, my client located in Colorado is opening a similar (but not identical) clinic in California. Will Google penalize the new California site if we use text from our website that features his Colorado office? He runs the clinic in CO and will be a partner of the clinic in CA, so the CA clinic has his "permission" to use his original text. Eventually he hopes to go national, with multiple sites utilizing essentially the same text. Will Google penalize the new CA site for plagiarism and/or duplicate content? Or is there a way to tell Google, "hey Google, this new clinic is not ripping off my text"?
Web Design | | CalamityJane770 -
Panda and Penquin Fall - Could HTML Design an Issue?
Hi, We were hit hard by Panda 3.4 on March 23rd 2012. Then Penguin came along and slapped us down a little farther on April 24th. White hat SEO for 13 years on the site. I have been trying to discover the reason we got hit so hard, to date 90% down. We ae wiped. I have a couple of keywords still #2 and #3 and we see up and down changes in Google webmaster tools, i.e. a keyword is supposedly up 50 points then another down 50. All other 150 keywords that we used to rank on the first page for are not even showing up. I have a person that is about to do a full link analysis but since we never went after links I just never had the feeling that is where our problem is at, but definitely going to explore it. The reason for my post is that last night I spoke with an SEO person that has some pretty good credentials (9 years experience and works currently at large online marketing company with seo with clients like Honda) and he was nice enough to just take a quick look at the site. He said he saw nothing really wrong and did not think that we were hit for any of the normal issues people are listing, i.e. duplicate content, backlinks. His first impression was that we were knocked down because the site is "hard to index". He said the site still uses tables and a lot of our Doc Statements were for HTML 4.01 from 1999. As we all know, there are 'many' experts in this industry. So I wanted a little feedback from the community. Our main site was built in Dreamweaver using tables. We do have a Wordpress blog that is very small and just now posting to add fresh content. (posts seem to rank pretty good, this is why I thought, you know he may be right) Would an older site be penalized like this for using tables? What would you do at this stage if you had a site that is not recovering? I have now reached panic mode and have to do something, just not sure of the next step. I will be happy to post the URL if anyone wants to help with advice. Thanks,
Web Design | | Force7
Force70 -
Duplicate content
Please help me to solve a problem with duplicate content errors.
Web Design | | NadiaFL
I received report today that 53 pages (all my pages) are market as duplicate
content. Yes, they are all about Oasis of the Seas but about different aspects
of it. Question is: how I can handle this error in report? Thank you!0 -
Given the lastest Google update, should I rewrite my Flash site or try to present an alternative HTML/CSS site?
I have a site that was created using Flash. The reasoning behind this was, at the time, that I didn't care if the site ranked or not (portfolio site). Now I would like to drive traffic to the site from SE's. Given the Penguin update, should I rewrite my Flash site in HTML/CSS or present an alternative site for bots and browsers that don't support Flash? My concern is that by presenting an alternative site to bots and non Flash supporting browsers that the SE's will see potentially see this as cloaking. Thoughts and advice would be much appreciated.
Web Design | | mj7750 -
Using "#" anchors to display different content
If I have a page that has an area on the page that acts like a widget and has three different tabs. These tabs provide 3 different types of information relevant to the page subject matter. By default when someone goes to the page one of the tabs is showing but you have to click on the others to see the info on them. Is it OK to use domain.com/topic#TAB1, domain.com/topic#TAB2, domain.com/topic#TAB3 to create shortcut links so that people can land on the page and have that predetermined tab showing. I'm wondering what search engines might think. Essentially all the content of all three tabs is there for people to see but they'd have to click to see the other tabs. I don't consider the content to be hidden. But I'd like to hear people's thoughts.
Web Design | | Business.com0 -
How do you get rid of the .html and .php extensions at the end of urls?
What is the whitehat way to properly remove the .html and .php extensions at the end of urls? Example: http://www.seomoz.org/learn-seo.php should be (and is) http://www.seomoz.org/learn-seo
Web Design | | Ryan-Bradley0 -
Does listing my customer's address, phone number, and a contact form on "every page" count as duplicate content that they'd be penalized for?
I work with small local businesses (like Tree Farms, Feed Stores, Counselors, etc) doing web design, seo, etc. I encourage them to have their contact information visible at all times on their websites. I'm also delving into the world of contact forms. I want to have this info on every page - is this detrimental? Here's an example: http://www.trinityescape.net/marriage-couples-counselors-therapy-clermont-florida/ Thank you!
Web Design | | mikjgens1