Moz crawler only crawls one page?!
-
Hello there,
I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled.
We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working.
Hopefully someone can point us at the right direction.
Thanks in advance,
Jeremy
-
Hey Jeremy,
I took a look at your campaign and it looks like Lynn's suggestion that Javascript may be blocking the crawl is correct. Unfortunately, our crawler isn't sophisticated enough to parse Javascript and all of the links from the homepage are hidden within the Javascript navigation so we aren't able to find them to crawl past the homepage. I'm afraid it looks like our crawler isn't compatible with this particular site.
I apologize for the inconvenience this causes! Please let me know if I can help you with anything further.
-
Thanks!!! I will take a look at the dev blog post
-
Hi,
It can happen for a number of reasons, in my experience usually due to either unintentional blocking of the page from crawling or from being a JavaScript heavy/only site which doesn't allow content to be easily read by the moz crawler. Check out this dev blog post for a rundown on most issues and solutions. Hope it helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Top pages according Link Explorer shows 301 pages and images
I was looking at the top pages on my website using Link Explorer, it contains 301 pages and image URLs. Is this expected or I should be doing something to fix it? URL with second highest PA in site is a page already 301 redirected more than 6 months back. URL with highest PA in my blog is a random image in the blog, rather than any blog posts. Is this normal? Thanks -Aji
Moz Bar | | ajiabs1 -
Significant difference with DA scores on Moz Chrome app VS Website Tool
I saw a big difference between DA scores on Moz Chrome app VS Website Tool. Which DA score is the correct one? I personally believe the scores on Moz's site is most accurate. Do you happen to have issues syncing the scored to your Chrome app? Chrome App VS. Moz Research Tool (https://moz.com/researchtools/ose/)
Moz Bar | | iPrice_Marketing
Moz.com: 92 / 88 Here's a screenshot of the difference: https://ipricegroup-my.sharepoint.com/:i:/p/jeremy_chew/ESy9lzUTC3lOl8o93Sx6LsUB8mYXo8LmYuUj2aa0xLXi1A?e=A6OrXs This was evident in other websites too:
Priceza (priceza.com.my😞 38 / 24
Shopback (shopback.my😞 43 / 41
Cuponation (https://www.cuponation.com.my/😞 27 / 250 -
What does it mean when Moz KW explorer returns a negative keyword difficulty score (eg -2)?
I recently did a keyowrd difficulty lookup in Moz keyword explorer as usual and for the first time a saw several (out of 100s) of negative keyowrd scores (generally -2). Is this a bug, what does it mean?
Moz Bar | | FlagshipCons1 -
On page grader tool - ip address
Does anyone know Moz's IP address for the above tool so I can give it access to a staging area of our site?
Moz Bar | | Jennie050 -
Can the Moz tool identify variations of a Chinese language branded keyword?
We've recently started a trial of the Moz Pro service and are tracking a selection of keywords. Our primary band / product name is 功夫英语(Kungfu English), so we've set a rule that any keywords containing those four Chinese characters (功夫英语) should be marked as a branded keyword. However, in the "Non-Paid Keywords Sending Search Visits" section of our traffic report, we see a few variations on our brand name that are not being marked as branded keywords. (See attached Images). Based on our rules, shouldn't these variations also be marked as branded keywords without our needing to manually add them as such? Or have I misunderstood the intent of this rule? For the English text, the brand rule about words containing "kungfuenglish" seems to have resulted in all of "www.kungfuenglish.com", "http://www.kungfuenglish.com", and kungfuenglish being labelled as branded keywords. However, I'm not seeing the same sort of result with variations on the Chinese keyword, 功夫英语. KedTi1D.png z5bcUIt.png
Moz Bar | | PaulCoffey0