Moz not crawling opencart product pages
-
hi,
i have waited for over 2 weeks now and the crawler only got 8 pages, and is not getting all the open cart pages and products.
any idea of what can be wrong?
im using joomla 2.5.11 and mijoshop 2.0.5 (which uses opencart 1.5.5.1).
thanks
-
Hi Marlon,
There are a lot of different issues that may cause Rogerbot to be unable to crawl your site. For troubleshooting tips, I suggest reading http://moz.com/help/pro/why-can-t-rogerbot-crawl-my-site. Hope that helps!
-
ok thanks
-
Screaming Frog has an option in the menu -
Configuration > Spider > "Ignore Robots.txt"
Tick this box and re-scan and see if you get different results.
And also, may be worth checking this out:
http://moz.com/learn/seo/robotstxtGood luck!
-
hi
thanks for the answer, tried screaming frog now as well and seems it has the same problem,
is it possible to allow only certain spiders/robots to crawl all files, with the robots.txt?
thanks
-
It's hard to say without knowing the domain name -
Have you tried Screaming Frog software to see if it also has issues crawling?
Double check your robots.txt to see if some directories and pages are blocked.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Pro OnDemand Crawl fail on on WordPress site
Hello, I just can't seem to understand why OnDemand Crawl fails on further attempts only 4 pages out of 68 I am using WordPress, Divi Theme and on LiteSpeed server. Robots.txt allows rogerbot just can seem to find the issue
Moz Bar | | ChrisSanClaire0 -
Page Grader states "includes Canonical Tag" but it's not in the page source at all
I've ran it multiple times and changed other things it picked up on so not sure where it's getting the canonical tag is included even though it isn't?
Moz Bar | | Wana-Ryd0 -
Is MOZ any good to analyze an e-commerce site? How come that a cms page can be seen as duplicate content with a category page?
Hi Guys, I've been using Moz for quite a long time now for 2 of my shops. Now I am in the process of launching the second shop and I just don't understand how is it possible that a cms static page (About US) to be seen as a duplicate content with other 96 pages - including product pages and other totally different pages such as delivery information, category pages, returns and so on. Really MOZ?? Is it me or you?? Your help would be much appreciated! Thank you!
Moz Bar | | Sorin_T0 -
How to find a list of pages with missing H1 tags
An external SEO/PPC agency did an audit of our site a little while back and said that over 10% of our pages were missing an H1 tag. I am trying to find a way to gather a full list of these in order for our web company to fix. I downloaded the Crawl Test report thinking it would include info on tags in there but it doesn't seem to. Is there a different tool I can use that will get me this information?
Moz Bar | | Lepra0 -
All in favor of an "Announcements" category section say MOZ!
So this is just a thought. I am not Rand, therefor I am not the Moz man himself, and therefor I am not telling anybody how to run this website or forum. But, I've been thinking... and usually that a dangerous thing. So many times when I answer a question I end it with: "Keep me posted" "Keep us updated" "Good Luck" In the same light there have been plenty of times when I have had my questions answered by people like EGOL, RobertFIsher, KeriMorget and they have said that they have wanted me to keep them posted. Now sure, some of you are sitting there being Melancholy Mozzers thinking that all those wonderful people are just being polite by adding some nice parting words... but what if I they are being Sincere Seoers? What if I WANT to fill them in and don't want to ask a question? What if... THEY REALLY LIKE ME?!?! I could also send a DM to those 3 people, I mean let's get honest I'm not THAT popular.... but I can dream can't I? Again, this isn't really an SEO question, more of a thought on where I'm supposed to give updates on my project. In all honesty I do want feedback on what I'm doing, and sometimes I don't even know what questions I'm supposed to ask!
Moz Bar | | HashtagHustler1 -
Probs with campaign overview page ??
Hi As of today my overview pages are devoid of formatting and just text and basic display, anyone else getting this ? Cheers Dan
Moz Bar | | Dan-Lawrence1 -
Why moz does not update actual keyword possition
I managed to get first place on google by keyword: vervo.lv kravu pārvadājumi in google.lv (it is in first place all this week) But in moz reports it shows just second position? I added so far about 100 keywords to monitor so if it shows incorrect this one how can i trust to other keyword results. Report says it updated on 3. september and it already was in first place. I really need to monitor keywords but if it does not show actual positions it is useless.
Moz Bar | | vervo0 -
Duplicate Page Content Report on MOZ
Hi, I am just wondering as to the accuracy of this report - does it pick up all the duplicate on page content? Or is there a limit? We have an ecommerce store with a lot of copied and pasted descriptions - just wondering if there is a limit on how much the moz crawler picks up? In other words, once we fix what MOZ has detected, will there be more detected because it is limited to display say up to 200?? Hope you understand what I mean. Thanks
Moz Bar | | bjs20100