Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Moz crawler only crawls one page?!
-
Hello there,
I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled.
We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working.
Hopefully someone can point us at the right direction.
Thanks in advance,
Jeremy
-
Hey Jeremy,
I took a look at your campaign and it looks like Lynn's suggestion that Javascript may be blocking the crawl is correct. Unfortunately, our crawler isn't sophisticated enough to parse Javascript and all of the links from the homepage are hidden within the Javascript navigation so we aren't able to find them to crawl past the homepage. I'm afraid it looks like our crawler isn't compatible with this particular site.
I apologize for the inconvenience this causes! Please let me know if I can help you with anything further.
-
Thanks!!! I will take a look at the dev blog post
-
Hi,
It can happen for a number of reasons, in my experience usually due to either unintentional blocking of the page from crawling or from being a JavaScript heavy/only site which doesn't allow content to be easily read by the moz crawler. Check out this dev blog post for a rundown on most issues and solutions. Hope it helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Looking for a Tool to Find Referring Pages of Specific URLs
Hello Everyone, We are Looking for a Tool to Find Referring Pages of Specific URLs. Please let me know if you know of a Moz tool or another tool for this need. Thanks.
Moz Bar | Jun 10, 2019, 6:19 AM | Pushm5 -
80% Spam Score!! Can Any One Help?
I have a question about the Spam Score for my website .
Moz Bar | Dec 5, 2018, 9:15 AM | PoppyP
I have run a Spam Score for https://www.poppyporter.co.uk
It has come back with an 80% score because there are 4 links that are apparently from my poppyporter.com domain each with a 69% spam score.
This poppyporter.com domain is parked and I have no idea why there are links from it. There are no pages there and there never have been, I've never used the poppyporter.com domain only poppyporter.co.uk.
I don't understand why there are links there and why they are making my Spam Score so high.
Will they affect my site's SEO? Do I need to worry about them? If so how do I get rid of them?
Can you help?
Thanks very much Poppy1 -
Limit MOZ crawl rate on Shopify or when you don't have access to robots.txt
Hello. I'm wondering if there is a way to control the crawl rate of MOZ on our site. It is hosted on Shopify which does not allow any kind of control over the robots.txt file to add a rule like this: User-Agent: rogerbot Crawl-Delay: 5 Due to this, we get a lot of 430 error codes -mainly on our products- and this certainly would prevent MOZ from getting the full picture of our shop. Can we rely on MOZ's data when critical pages are not being crawled due to 430 errors? Is there any alternative to fix this? Thanks
Moz Bar | Oct 3, 2018, 1:21 PM | AllAboutShapewear2 -
Moz tool bar shows "No markup schema" in webpage despites having proper schema code in page.
We have proper markup schema installed in webpage (validated using data structure tool by google) but moz tool bar says "Schema.org not found on this page." What should be the problem ?
Moz Bar | Aug 31, 2020, 1:12 PM | NortonSupportSEO1 -
Only One Canonical URL Tag
HI, I'm an SEO novice - company owner with no money so doing it all myself with help from my web designer using wordpress. Ive just completed some seo and done the moz page scoring analysis for optimisation and gained 92% - however - there is one outstanding issue on canonical url tags - i.e. recommened fix = The canonical URL tag is intended to refer duplicate pages to a single canonical URL. To ensure the search engines properly parse the canonical source, your page should use only one version of this tag in the header. See Canonical URL Tag - the Most Important Advancement in SEO Practices Since Sitemaps Ive gone through the page code and can see I have 2 rel=canonical references - am I able to simply delete one - how do I do this if its been created by the yoast/wordpress plug-in? Many thanks in advance for any help!
Moz Bar | Mar 24, 2017, 4:11 AM | M-J-Smith0 -
Moz reporting for C-Blocking
Hey Mozers, I see Moz has a reporting tool for C-blocking and for november I had 330. Does this mean 330 Ip addresses came from the same location in the month of november?
Moz Bar | Jan 2, 2015, 1:44 PM | rpaiva1 -
Duplicate Page and Title Issues
On the last crawl, we received errors for duplicate page titles and some duplicate content pages. Here is the issue: We went through our page titles that were marked as duplicate and changed them to make sure their titles were different. However, we just received a new crawl this week and it is saying there are even more duplicate page title errors detected than before. We're wondering if this is a problem with just us or if it has been happening to other Moz users. As for the duplicate content pages, what is the best way to approach this and see what content is being looked at as a "duplicate" set?
Moz Bar | Apr 29, 2020, 9:09 PM | Essential-Pest0
