Seomoz crawling filtered pages
-
Hi,
I just checked an seo campaign we started last week, so I opened seomoz to see the crawl diagnostics.
Lot's of duplicate content & duplicate titles showing up, but that's because Rogerbot is crawling all of the filtered pages as well. How do I exclude these pages from being crawled?
/product/brand-x/3969?order=brand&sortorder=ASC
/product/brand-x/3969?order=popular&sortorder=ASC
/product/brand-x/3969?order=popular&sortorder=DESC&page=10
/product/brand-x/3969?order=popular&sortorder=DESC&page=11 -
So if the site has a structure like this:
and the filter on this page has several options like /?order= , ?brand=, .... I have to rel-canonical them al to www.xyz.com/overview
-
I'd rel-canonical if you can, as theres still nothing stopping links to them being indexed. It might stop Rodger/Google from crawling them, but the potential indexation issues won't go away. Otherwise perhaps no-index them.
I'd usualy go as far to do re-prev and rel-next for the paginated searches as well.
-
Rel-canonical will be an immense job, would it be ok using robots.txt like this
disallow: /?order=
disallow: /?sortorder=
...
?
-
Last I checked, Rodger obays robot.txt and meta-robot commands as he trys to simulate what google would crawl. If he can crawl those pages, google probably is as well.
I think if you rel-canonical them properly, or no-idex them properly e.t.c. It should show as a normal page.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Pages with Temporary Redirects on pages that don't exist!
Hi There Another obvious question to some I hope. I ran my first report using the Moz crawler and I have a bunch of pages with temporary redirects as a medium level issue showing up. Trouble is the pages don't exist so they are being redirected to my custom 404 page. So for example I have a URL in the report being called up from lord only knows where!: www.domain.com/pdf/home.aspx This doesn't exist, I have only 1 home.aspx page and it's in the root directory! but it is giving a temp redirect to my 404 page as I would expect but that then leads to a MOZ error as outlined. So basically you could randomize any url up and it would give this error so I am trying to work out how I deal with it before Google starts to notice or before a competitor starts to throw all kinds at my site generating these errors. Any steering on this would be much appreciated!
Moz Pro | | Raptor-crew0 -
How to increase page authority
I wonder how to increase the page authority or the domain authority to begin with. It seems you are putting a lot of weight on this in your analysis.
Moz Pro | | wcsinc0 -
Are seomoz ranking in jeopardy?
A few weeks ago Raventools announced that they would no longer be able to provide tools based on the adwords API, then withdrew that. Rand commented on a thread here discussing that and made similar noises. Today Raventools made an announcement as SERP tracking with wording that sounded a lot like it had more than a little influence from Google. Would I be really paranoid to be wondering whether the Rankings tools in seomoz could be at risk?
Moz Pro | | matbennett1 -
Does SEOmoz give a way to know what link on what page produces the 404 errors that SEOmoz is telling me I have??
SEOmoz gives me a report of 404 errors on my site. Do they give a way to know from what link on what page produces the error?
Moz Pro | | MeridianGroup0 -
How to resolve Duplicate Content crawl errors for Magento Login Page
I am using the Magento shopping cart, and 99% of my duplicate content errors come from the login page. The URL looks like: http://www.site.com/customer/account/login/referer/aHR0cDovL3d3dy5tbW1zcGVjaW9zYS5jb20vcmV2aWV3L3Byb2R1Y3QvbGlzdC9pZC8xOTYvY2F0ZWdvcnkvNC8jcmV2aWV3LWZvcm0%2C/ Or, the same url but with the long string different from the one above. This link is available at the top of every page in my site, but I have made sure to add "rel=nofollow" as an attribute to the link in every case (it is done easily by modifying the header links template). Is there something else I should be doing? Do I need to try to add canonical to the login page? If so, does anyone know how to do it using XML?
Moz Pro | | kdl01 -
Edit SEOmoz Profile Bug
Hi,
Moz Pro | | rayvensoft
I am trying to complete my SEOmoz profile, but there seems to be a bug. I am able to fill out everything except the 'favorite topics'. It lets me go in and fill it out, but when I click the update button to save it it does not save, and still shows up as missing in bar at the top. I have tried it in Mozilla, IE and Chrome, and it still does not work. Am I missing something? Thanks.0 -
Issue in number of pages crawled
i wanted to figure out how our friend Roger Bot works. On the first crawl of one of my large sites, the number of pages crawled stopped at 10000 (due to the restriction on the pro account). However after a few weeks, the number of pages crawled went down to about 5500. This number seemed to be a more accurate count of the pages on our site. Today, it seems that Roger Bot has completed another crawl and the number is up to 10000 again. I know there has been no downtime on our site, and the items that we fixed on our site did not reduce or increase the number of pages we had. Just making sure there are no known issues with Roger Bot before I look deeper into our site to see if there is an issue. Thanks!
Moz Pro | | cchhita0