Moz crawler only crawls one page?!
-
Hello there,
I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled.
We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working.
Hopefully someone can point us at the right direction.
Thanks in advance,
Jeremy
-
Hey Jeremy,
I took a look at your campaign and it looks like Lynn's suggestion that Javascript may be blocking the crawl is correct. Unfortunately, our crawler isn't sophisticated enough to parse Javascript and all of the links from the homepage are hidden within the Javascript navigation so we aren't able to find them to crawl past the homepage. I'm afraid it looks like our crawler isn't compatible with this particular site.
I apologize for the inconvenience this causes! Please let me know if I can help you with anything further.
-
Thanks!!! I will take a look at the dev blog post
-
Hi,
It can happen for a number of reasons, in my experience usually due to either unintentional blocking of the page from crawling or from being a JavaScript heavy/only site which doesn't allow content to be easily read by the moz crawler. Check out this dev blog post for a rundown on most issues and solutions. Hope it helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Is there a similar tool in Moz to SEM Rush Topic Research?
Re: Related topics / content suggestion Is there a similar tool in Moz that is like the SEM Rush Topic Research tool?
Moz Bar | | CAPTRUST0 -
Can Moz Keyword Explorer help target keywords for Google Images results?
I'm wondering if I can use Keyword Explorer (or maybe another tool?) to target keywords for image rankings. I'd like to play around with optimizing images so that they appear in search results and thus provide traffic - but wasn't sure the best way to track that kind of progress. My ultimate goal is to analyze the difficulty of ranking for a certain keyword via Google images. (I do know to optimize alt tag/title tag/place in relevant article etc, but wanted to know if I could research the difficulty). Any help is much appreciated. Thanks!
Moz Bar | | naturalsociety0 -
How do you use Moz to research related topics?
Like most of the folks here I'm a pretty big fan of the content that comes out through Whiteboard Fridays, and I try to apply the things I learn, but one of the WBF videos that I'm following along with does not do a stellar job of detailing execution using Moz KW Explorer. https://moz.com/blog/related-topics-in-seo-whiteboard-friday Now granted, this came out in 2016, but I still feel the core principle and strategy results in a higher quality piece of content and is still relevant to discovering and understanding searcher task completion requirements, and drafting content that fulfills those requirements. Towards the end Rand sort of mentions that you'll be able to do this with KW explorer, but I'm not really seeing the functionality. The steps I followed were to enter in the keyword in kw explorer, went to keyword suggestions, and selected "based on closely related topics" and ran it, but received no suggestions - came up blank. I then selected "based on broadly related topics" and the same thing happened. I tried this out with the keyword r22, keeping it very broad to start but that didn't seem to work. So what do you all do to perform this sort of research within Moz? Or do you even feel it's relevant in today's Rank Brain driven world?
Moz Bar | | brettmandoes0 -
Significant difference with DA scores on Moz Chrome app VS Website Tool
I saw a big difference between DA scores on Moz Chrome app VS Website Tool. Which DA score is the correct one? I personally believe the scores on Moz's site is most accurate. Do you happen to have issues syncing the scored to your Chrome app? Chrome App VS. Moz Research Tool (https://moz.com/researchtools/ose/)
Moz Bar | | iPrice_Marketing
Moz.com: 92 / 88 Here's a screenshot of the difference: https://ipricegroup-my.sharepoint.com/:i:/p/jeremy_chew/ESy9lzUTC3lOl8o93Sx6LsUB8mYXo8LmYuUj2aa0xLXi1A?e=A6OrXs This was evident in other websites too:
Priceza (priceza.com.my😞 38 / 24
Shopback (shopback.my😞 43 / 41
Cuponation (https://www.cuponation.com.my/😞 27 / 250 -
Only One Canonical URL Tag
HI, I'm an SEO novice - company owner with no money so doing it all myself with help from my web designer using wordpress. Ive just completed some seo and done the moz page scoring analysis for optimisation and gained 92% - however - there is one outstanding issue on canonical url tags - i.e. recommened fix = The canonical URL tag is intended to refer duplicate pages to a single canonical URL. To ensure the search engines properly parse the canonical source, your page should use only one version of this tag in the header. See Canonical URL Tag - the Most Important Advancement in SEO Practices Since Sitemaps Ive gone through the page code and can see I have 2 rel=canonical references - am I able to simply delete one - how do I do this if its been created by the yoast/wordpress plug-in? Many thanks in advance for any help!
Moz Bar | | M-J-Smith0 -
External Links not showing in MOZ?
Hello:) I have a ton of external links on my site, but there not showing on MOZ? The reason i'm slightly concerned is because my top competitor has 80 and I show as only having 3? Does any one know why this would be? Also, my traffic on MOZ is a lot lower then my traffic report on Google Analytics? Is that normal, & which should I trust more? Here's my Site: Www.ThumannAgency.com Thank you in advance:)
Moz Bar | | MissThumann0 -
Moz Content --- for SEO or simply user engagement?
What is the primary function of Moz Content? It looks like it is most useful for managing content as a user engagement tool. Our content strategy is centered on boosting organic placement - with user engagement as a nice but unessential side product. Besides providing general descriptive details of a site's content / authorship - how can Moz Content help with SEO?
Moz Bar | | cvonhassell0 -
URL inaccessible for On Page Grader
I am trying to use the on page grader however it is not working with my website. The URL is as follows: https://capbeast.com. I have been trying to read in older posts to see if https is now supported or not but have not found anything. I know there is no robots.txt issue as I am able to run the crawl test on our website fine. Is the issue on my end in regards to configuration or is due to DDos attacks? Any help would be appreciated. Thanks
Moz Bar | | MisterStitches0