Domain.com and domain.com/index.html duplicate content in reports even with rewrite on
-
I have a site that was recently hit by the Google penguin update and dropped a page back. When running the site through seomoz tools, I keep getting duplicate content in the reports for domain.com and domain.com/index.html, even though I have a 301 rewrite condition. When I test the site, domain.com/index.html redirects to domain.com for all directories and root. I don't understand how my index page can still get flagged as duplicate content.
I also have a redirect from domain.com to www.domain.com.
Is there anything else I need to do or add to my htaccess file?
Appreciate any clarification on this.
-
Hello Anthony,
Saw this still open.
If your index.html "Rewrite" code is accurate, could the issue be WWW, i.e. http://www.domain.com vs. http://domain.com?
RewriteCond %{HTTP_HOST} ^domain.com
RewriteRule ^(.*)$ http://www.domain.com/$1 [R=permanent,L] -
I checked one of your campaigns, and it does seem like the 301-redirect is working properly. I'm also not seeing any evidence of links to the "index.htm" version or other issues. I don't see evidence of both version sin Google's index. Not sure exactly what's going on here, but I'll run it by the support team. I don't think you have cause for concern.
-
Thank you for the feedback and help.
I have looked up url removal in webmaster tools and it states that the page must be removed from the site. If I remove index.html I wont have a home page. Am I understanding you correctly? Heres what google states on url removal.
To remove a page or image, you must do one of the following:
- Make sure the content is no longer live on the web. Requests for the page must return an HTTP 404 (not found) or 410 status code.
- Block the content using a robots.txt file.
- Block the content using a meta noindex tag.
Please clarify when you get a moment.
I would have thought the htaccess 301 redirects from www.domain.com/index.html to www.domain.com would be enough.
Thank you in advance.
-
a) request removal of the /index.html URL in webmaster tools and it will go away in Google's index quickly.
b) make sure that when you link to your homepage on your site you are not linking to the /index.html URL - I bet you are somewhere do a sitewide search in dreamweaver to find all instances and do a global replace.
-
It could take a little time. I did some redirects myself earlier this year, but the old pages are still in Google's index.
Maybe someone else can confirm that it can take a little time before the old pages are dropped from Google's index?
-
HTTP/1.1 301 Moved Permanently => Date => Tue, 08 May 2012 13:44:26 GMT Server => Apache/2.0.52 (CentOS) Location => http://www.domain.com/ Content-Length => 330 Connection => close Content-Type => text/html; charset=iso-8859-1
-
Did you verify with a tool like http://www.webconfs.com/http-header-check.php that you get a 301 redirect?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Domain still not being found in search
Hi guys, I've been handed a client who needs some seo work. I've tweaked one of their pages to focus on a chosen keywords about 4 months back but still the site is not even visible using the new Domain Analysis tool from moz and it still won't rank at all for the keywords. Am I missing something here? Is there something blocking the SERP from listing the website? I've ran a site: search on Google and it returns 283 results on the website. It's puzzling me as there clearly is something stopping it from being ranked. The domain name in question is: https://cloud9inecommunications.co.uk Thanks in advance.
Moz Pro | | Easigrass1 -
Difference between Open site Explorer's Root Domain and Basic SERP Report's Linking Root Domain?
Why show different Linking Root Domain open site explorer and SERP of any websites? Open Site explorer show different linking root domain and Basic SERP Report show different linking root domain of any website url, who is the correct and why it is show different linking root domain?
Moz Pro | | surabhi60 -
Why would SEOmoz be blocked from a domain?
Am I missing something? I'm trying to do Link Research on www.weddingcarsofhampshire.co.uk but it seems to be blocked, any ideas why that would be the case?
Moz Pro | | JemRobinson0 -
Confused - SEOMoz Reporting On Different Domains?
I wasn't sure which report to set up on my domain, so I set up 2 different ones which now appear like this in the Dashboard under the Campaigns section: *mysite.com www.mysite.com I want to delete the wrong report, but I noticed that they produce two slightly different report results which is interesting. My domain does not have any subdomains - which report should I delete?
Moz Pro | | Ubique0 -
Finding the source of duplicate content URL's
We have a website that displays a number of products. The product has variations (sizes) and unfortunately every size has its own URL (for now anyway). Needless to say, this causes duplicate content issues. (And of course, we are looking to change the URL's for our site as soon as possible) However, even though these duplicate URL's exist, you should not be able to land on them by navigating through the site. In theory, the site should always display the link to the smallest size. It seems that there is a flaw in our system somewhere, as these links are now found in our campaign here on SEOmoz. My question: is there any way to find the crawl path that lead to the URL's that shouldn't have been found, so we can locate the problem?
Moz Pro | | DocdataCommerce0 -
Issue: Duplicate page title
Hello, I have run the "Crawl Diagnostics" report using SEOmoz pro and it says that I have a total of 56 errors. 18 of those errors being duplicate content and another 38 errors being duplicate title tags. Now I have looked at both reports and detail and the reason I am getting there errors is due to the fact the it is checking "http" and "https". So for example: my website is http://www.widgets.com On the crawl diagnostics report, it also checks https://www.widgets.com So it looks like I have duplicate content and duplicate title tags because of this Now my question is this: Is this really duplicate content? If so, how do I fix this? Any help is greatly appreciated.
Moz Pro | | threebiz0 -
Domain Authority is up, how do I find out what changed?
Hi, I’m a noob so if this is a really obvious question, I apologize. Our domain authority increased this week. I wonder if there is any easy way to find out what changed. (We didn’t do anything.) Thank you! Ann
Moz Pro | | anns0 -
Domain Name Portfolio Management?
I am looking for some software, or an excel spreadsheet, or some way to manage 50-100 domain names. Maybe something a reseller of domain names would use? Something that keeps record of different whois information for different domain names, tracks IPs, nameservers, FTP logins, etc... I think excel would be best for this if there was a way to have the expiration date pulled so I dont forget to renew a domain name by mistake, and so I can easily keep track of them all in one place. I have seen a couple software programs out there for this, but none really meet all the needs I have. I also looked into some kinda database software with a nice GUI (MS Access, or FileMakerPro) but I dont know them software well. Any Ideas?
Moz Pro | | getbigyadig0