Crawl Diagnostics returning duplicate content based on session id
-
I'm just starting to dig into crawl diagnostics and it is returning quite a few errors. Primarily, the crawl is indicating duplicate content (page titles, meta tags, etc), because of a session id in the URL.
I have set-up a URL parameter in Google Webmaster Tools to help Google recognize the existence of this session id. Is there any way to tell the SEOMoz spider the same thing? I'd like to get rid of these errors since I've already handled them for the most part.
-
You the man! Thanks!
-
Hi Cody,
The best way is to block Rogerbot within your Robots.txt from crawling specific pages of your site. In your case protecting Rogerbot from seeing the pages with a session ID.
More information could be found here on Rogerbot.Be cautious and test it out, but the lines you would have to add to your Robots.txt are probably:
User-agent: rogerbot
Disallow: /*sessionidHope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Shopify crawl issues
Hi Moz'ers, I am a total newcomer to this level of seo. Recently I transitioned to Shopify and I'm puzzled by why I'm getting 803 errors - incomplete crawl attempts due to server timing out. Wouldn't this have to do with Shopify? How would I go about fixing it? I'm also getting 804 - SSL issues, but I assume that will go away. Any advice? Thanks! Sharon
Moz Pro | | Sharon2016
www.ZeldasSong.com0 -
To Worry or Not? Duplicate Content Created from Redirect After Login
One of my Moz reports is flagging duplicate content. For example, https://redchairmarket.com/Account/LogOn?ReturnUrl=%2FAccount%2FSaveSearch%3FsearchId%3D0&searchId=0 and https://redchairmarket.com/Account/LogOn?ReturnUrl=%2FAccount%2FSaveSearch%3FsearchId%3D1&searchId=1 are created when a user logs in and the website sends them back to the page they were looking at before. What is the best way to deal with this duplicate issue? How serious is it? Thank you!
Moz Pro | | BrittanyHighland0 -
Are there tools to discover duplicate content issues with the other websites?
We have issues with users copy-pasting content from other sources into our site. The only way I know to find out, is to manually (!!) copy a snippet of their text into google, to see if I get results from other sites. I have been googling for tools to help automate this process, but without luck. Can you recommend any?
Moz Pro | | betternow0 -
Crawl Diagnostics - Historical Summary
As we've been fixing errors on our website, the crawl diagnostic graphs have been showing great results (top left to bottom right for errors). The problem is the graphs themselves aren't very pretty. I can't use them in my internal reports (all internal reports are standardised colours/formats). Is there anyway of exporting the top level summary with historic data so the graphs can be recreated in company colours? I don't want the detailed CSV breakdown of what errors occurred, but rather than on X date there were Y errors, the next month Z errors and so forth. The data must already be in the SEOMoz system in order to create the graphs themselves - I was hoping this can be made available to us if it isn't already? Does anyone know if there is already a way of doing this? I've tried to 'inspect element' and find the underlying data in the source code but to no avail, and can't see any exports that would do this. Thanks in advance Dean
Moz Pro | | FashionLux0 -
When I did my first crawl, I was given some errors.
Do I then need to re-crawl to make sure the errors were fixed accordingly?
Moz Pro | | immortalgamer0 -
Why does Crawl Diagnostics report this as duplicate content?
Hi guys, we've been addressing a duplicate content problem on our site over the past few weeks. Lately, we've implemented rel canonical tags in various parts of our ecommerce store, over time, and observing the effects by both tracking changes in SEOMoz and Websmater tools. Although our duplicate content errors are definitely decreasing, I can't help but wonder why some URLs are still being flagged with duplicate content by our SEOmoz crawler. Here's an example, taken directly from our Crawl Diagnostics Report: URL with 4 Duplicate Content errors:
Moz Pro | | yacpro13
/safety-lights.html Duplicate content URLs:
/safety-lights.html ?cat=78&price=-100
/safety-lights.html?cat=78&dir=desc&order=position /safety-lights.html?cat=78 /safety-lights.html?manufacturer=514 What I don't understand, is all of the URLS with URL parameters have a rel canonical tag pointing to the 'real' URL
/safety-lights.html So why is SEOMoz crawler still flagging this as duplicate content?0 -
Duplicate content due to "Email a Friend" and "PhotoGallery"
Crawl Diagnostics gives me 1650 duplicate page content errors. 800+ are for photo gallery, the link upon which my large image is shown. 800+ are for the Email a Friend form page. Presumably SEOMoz's tool is getting there by following the link inside a product page, which as you can see will differ only by the ProductCode of every product.
Moz Pro | | jotham2
www.completemobilehomesupply.com/PhotoGallery.asp?ProductCode=anchor101
www.completemobilehomesupply.com/EmailaFriend.asp?ProductCode=Shutter001 1 - Is it critical to fix?
I am assuming Yes, but if for some reason the answer is No, please share. 2 - Any idea on how to fix?
The site uses Volusion, fyi, so it may be a limiting factor of the platform. Thank you for your time.
Also if this topic has been previously covered, please link and I'll read there instead.0 -
Help with duplicate title tags?
I was looking in Google webmaster tools and it says I have 95 duplicate title tags for my site Noah's Dad. When I look through the list it appears the pages with duplicate title tags are some of my category pages, archive pages, and some author pages... Not sure if you guys can use some of the tools to see what is actually showing up duplicate or not, and if you need more info just let me know. But I wanted to see if this is something I should be concerned with? Should WMT also say 0 in duplicate content? It seems like when I started my blog I was told no to be conceded with this sort of stuff in gwmt. Anyways...I just wanted to see what you guys think. (By the way, is there any way to tell what this duplicate content is having (or has had) on my SERP results? Thanks.
Moz Pro | | NoahsDad0