Why isn't the Moz crawler getting all of my item pages?
-
I am stumped and Moz is being terrible to work with.
This site has about 40k pages
39,800 of them are item pages roughly.
Moz is only finding about 2400 of my pages. So they are missing most but not all of my item pages.
I do not know which item pages they are missing.
The fact that they are finding about 2k but not the rest leads me to believe the crawler is struggling with pagination.
The site is built on Magento 2 and uses the Amasty Layered Navigation extension.
Does anyone have any ideas?
-
Meghan from Moz pointed out I had Disallow: /*? in the robots.txt and that appears to have been the problem. It is the default Magento 2 setting, so not sure why they had that in there.
-
Hi Tyler, are you setting your pagination pages as nofollow? That might be causing the issue. Any chance you can post the URL and we can try crawling the site to see if we hit any issues with a crawlers like screaming frog?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is MOZ any good to analyze an e-commerce site? How come that a cms page can be seen as duplicate content with a category page?
Hi Guys, I've been using Moz for quite a long time now for 2 of my shops. Now I am in the process of launching the second shop and I just don't understand how is it possible that a cms static page (About US) to be seen as a duplicate content with other 96 pages - including product pages and other totally different pages such as delivery information, category pages, returns and so on. Really MOZ?? Is it me or you?? Your help would be much appreciated! Thank you!
Moz Bar | | Sorin_T0 -
The Old Moz Pro
Hello, I found the old moz pro a lot more informative for certain queries. Can I still access this and run a campaignMany Thanks
Moz Bar | | summer3000 -
How is Moz's DA score so high?
Obviously they make the tool that measures this, but is their own score inflated because of this? I know DA isn't strictly a measure of popularity, but Moz's own score seems a little _too _high compared to some other sites... 92 - Moz.com 90 - Salesforce.com 87 - Airbnb.com 84 - Priceline.com 83 - Squarespace.com Can someone shed some light on how DA is actually measured?
Moz Bar | | WickVideo0 -
Idea for new feature on Moz Q&A
Hi Moz Can we please have a new feature to moz when answering questions please, the ability to save as a draft. I am currently on the train from London back home and I was trying to answer a few questions and give my opinion and after writing a really long and detailed response to an area I actually have quite a lot to say on, my internet froze and the page refreshed losing all the content. It doesn’t help that my Mac I am working on isn’t the fastest, but my new one isn’t quite ready yet. But if there was a simple save draft feature, it would be really great as I wasn’t ready to post my answer as it wasn’t finished and without checking wouldn’t have made sense. Yes I know I could of done it in word and pasted it across, but usually when I am travelling and answering questions I do it on my Ipad which isn’t great for multitasking. While I understand you don’t want lots of ‘draft answers’ all over the place as this would take storage space up on your servers and doesn’t actually help the person answering the question, your system could purge all draft answers which have not been edited in the last 24 hours. Just a little suggestion to help make this great forum even better. Thanks Andy
Moz Bar | | Andy-Halliday4 -
My product pages have no weight / links from root domains with the Moz tool bar
Hi, When I view my home page (http://www.arkwildlife.co.uk) with the Moz toolbar, I see a good PA and a good amount of links from root domains. As I go down the site, it seems to get worse. The category pages (http://www.arkwildlife.co.uk/Category/0/Straight_Foods.html) have a little but not much and then from this point onwards, it's nothing. The product page (http://www.arkwildlife.co.uk/Item/Straight_Foods~Sunflower_Seeds/SUNH/Premium_Sunflower_Hearts.html) is reporting to have no root domain links but I am not sure why. Interestingly, when I click through to a review page (http://www.arkwildlife.co.uk/StockReview/0/SUNH/0/Premium_Sunflower_Hearts.html) it does have some juice. Would anyone be able help on why this is happening and what I need to be looking at in order to resolve it please? EDIT: I've been looking at the hyperlinks and notice something odd. If I review the score with the first link below, it gives a score of 1, but the second gives a PA of 13 with one root domain linked. 1:http://www.arkwildlife.co.uk/Item/Straight_Foods~Sunflower_Seeds/SUNH/Premium_Sunflower_Hearts.html 2:http://www.arkwildlife.co.uk/Item/Straight_Foods%7ESunflower_Seeds/SUNH/Premium_Sunflower_Hearts.html Please note the "%7E" instead of the "~" in the URL. The browser appears to show the ~ character no matter what but the rank of the page changes. I don't understand what the Moz toolbar is doing with this. Note: This behaviour only happens in Mozilla Firefox, in chrome both the rankings are zero for each URL. Many Thanks.
Moz Bar | | nawgie0 -
Dupe content report showing in 'Errors' section when surely should be in 'Warnings' section ?
Why is the dupe content info showing in errors and not warnings ? Since if dupe content can get your site penalised (as per Panda) or worse banned, surely it should be in that section of reports ? Cheers
Moz Bar | | Dan-Lawrence
Dan0 -
Moz not crawling opencart product pages
hi, i have waited for over 2 weeks now and the crawler only got 8 pages, and is not getting all the open cart pages and products. any idea of what can be wrong? im using joomla 2.5.11 and mijoshop 2.0.5 (which uses opencart 1.5.5.1). thanks
Moz Bar | | marlvass10