Domain.com and domain.com/index.html duplicate content in reports even with rewrite on
-
I have a site that was recently hit by the Google penguin update and dropped a page back. When running the site through seomoz tools, I keep getting duplicate content in the reports for domain.com and domain.com/index.html, even though I have a 301 rewrite condition. When I test the site, domain.com/index.html redirects to domain.com for all directories and root. I don't understand how my index page can still get flagged as duplicate content.
I also have a redirect from domain.com to www.domain.com.
Is there anything else I need to do or add to my htaccess file?
Appreciate any clarification on this.
-
Hello Anthony,
Saw this still open.
If your index.html "Rewrite" code is accurate, could the issue be WWW, i.e. http://www.domain.com vs. http://domain.com?
RewriteCond %{HTTP_HOST} ^domain.com
RewriteRule ^(.*)$ http://www.domain.com/$1 [R=permanent,L] -
I checked one of your campaigns, and it does seem like the 301-redirect is working properly. I'm also not seeing any evidence of links to the "index.htm" version or other issues. I don't see evidence of both version sin Google's index. Not sure exactly what's going on here, but I'll run it by the support team. I don't think you have cause for concern.
-
Thank you for the feedback and help.
I have looked up url removal in webmaster tools and it states that the page must be removed from the site. If I remove index.html I wont have a home page. Am I understanding you correctly? Heres what google states on url removal.
To remove a page or image, you must do one of the following:
- Make sure the content is no longer live on the web. Requests for the page must return an HTTP 404 (not found) or 410 status code.
- Block the content using a robots.txt file.
- Block the content using a meta noindex tag.
Please clarify when you get a moment.
I would have thought the htaccess 301 redirects from www.domain.com/index.html to www.domain.com would be enough.
Thank you in advance.
-
a) request removal of the /index.html URL in webmaster tools and it will go away in Google's index quickly.
b) make sure that when you link to your homepage on your site you are not linking to the /index.html URL - I bet you are somewhere do a sitewide search in dreamweaver to find all instances and do a global replace.
-
It could take a little time. I did some redirects myself earlier this year, but the old pages are still in Google's index.
Maybe someone else can confirm that it can take a little time before the old pages are dropped from Google's index?
-
HTTP/1.1 301 Moved Permanently => Date => Tue, 08 May 2012 13:44:26 GMT Server => Apache/2.0.52 (CentOS) Location => http://www.domain.com/ Content-Length => 330 Connection => close Content-Type => text/html; charset=iso-8859-1
-
Did you verify with a tool like http://www.webconfs.com/http-header-check.php that you get a 301 redirect?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What's more valuable: new content or optimizing old content
We are a niche legacy print publication that's been around for close to 20 years. Recently, we combined several old sites in one new responsive site. We have over 7,000 articles -many of which are evergreen and can be repurposed when needed. Most of the old pieces although published, have not been optimized for SEO. However, as we create new pieces, we optimize them for search and social and they tend to get more organic traffic. Where we're torn is on how much we should balance our limited editorial resources between cleaning up and optimizing our extensive archive to improve our organic reach, vs. pumping out new original pieces each week. I realize that without a lot of data the answers will be varied - I guess I'm looking for a best practices approach for content publishers. If it helps at all, our main conversion goal is selling subscriptions to our print and digital publications. We know that organic traffic tends to be more engaged than our social referrals. Unfortunately, due to the nature of the magazine fulfilment business, it's tough to know which channels convert better. Thanks!
Moz Pro | | RicardoSalcedo0 -
C-Block domains OSE
hi all quick question regarding c-block domains OSE tells me we have 70 c-block domains with a total 130 root domains, is it telling us 70 root domains re c-blocks as this is near impossible for us are c blocks listed as root domains or just links
Moz Pro | | Will_Craig0 -
Why has the Historical Domain Analysis report not updated since 11th July 2013?
I find the Historical Domain Analysis report within Moz really useful but it's stopped updating for some reason. Is anyone else seeing this? When can we next expect it to update?
Moz Pro | | TranslateMediaLtd0 -
Duplicate content in crawl despite canonical
Hi! I've had a bunch of duplicate content issues come up in a crawl, but a lot of them seem to have canonical tags implemented correctly. For example: http://www.alwayshobbies.com/brands/aztec-imports/-catg=Fireplaces http://www.alwayshobbies.com/brands/aztec-imports/-catg=Nursery http://www.alwayshobbies.com/brands/aztec-imports/-catg=Turntables http://www.alwayshobbies.com/brands/aztec-imports/-catg=Turntables?page=0 Aztec http://www.alwayshobbies.com/brands/aztec-imports/-catg=Turntables?page=1 Any ideas on what's happening here?
Moz Pro | | neooptic0 -
In alt tag of a image can we use #hashtag or domain.com ? Is that good SEO or not allowed ?
Some of the Google Search shows a title has a hashtag of an article, which contain keyword and while tweeting them, the title which has a hashtag automatically very good used for getting traffic to the blog. And other one, can we use the hash tag inside the alt attribute ? Or our domain name with .com in it. Like Google.com or #Google ?
Moz Pro | | Esaky0 -
I know our business listed in Yahoo and medranks.com (for example). But my open site explorer report doesn't show those. however on their sites, I see the listing. Why is this?
I know our business listed in Yahoo and medranks.com (for example). But my open site explorer report doesn't show those links on the inbound report. however on their respective sites, I see the listing when I search for us. And the link does work..... Why is this? Why don't I see it on the open site report?
Moz Pro | | cschwartzel0 -
How do I fix a duplicate content error with a top level domain?
Hi, I'm getting a duplicate content error from the SEOmoz crawler due to an issue with trailing slashes. It's showing www.milengo.com and www.milengo.com/ as having duplicate page titles. However I'm pretty sure this has been fixed in the .htaccess file since if you type in the domain with a trailing slash it automatically redirects to the domain without a trailing slash, so this shouldn't be an issue. I'm stuck here. Any ideas? Thanks. Rob
Moz Pro | | milengo0 -
Where is the labs LDA topics tool? The articles mentioning it point to http://www.seomoz.org/labs/lda but that forwards me to the tools landing page.
I searched SEOMOZ for this but haven't yet found the answer. It seems that it's been removed. I'm sure it was announced somewhere that I missed
Moz Pro | | bluenote0