Moz crawler only crawls one page?!
-
Hello there,
I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled.
We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working.
Hopefully someone can point us at the right direction.
Thanks in advance,
Jeremy
-
Hey Jeremy,
I took a look at your campaign and it looks like Lynn's suggestion that Javascript may be blocking the crawl is correct. Unfortunately, our crawler isn't sophisticated enough to parse Javascript and all of the links from the homepage are hidden within the Javascript navigation so we aren't able to find them to crawl past the homepage. I'm afraid it looks like our crawler isn't compatible with this particular site.
I apologize for the inconvenience this causes! Please let me know if I can help you with anything further.
-
Thanks!!! I will take a look at the dev blog post
-
Hi,
It can happen for a number of reasons, in my experience usually due to either unintentional blocking of the page from crawling or from being a JavaScript heavy/only site which doesn't allow content to be easily read by the moz crawler. Check out this dev blog post for a rundown on most issues and solutions. Hope it helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can a page have high Google/ organic traffic but show no ranking keywords in Moz?
We have a page on our website with a higher than average number of pageviews, 85% of which came from Google organic search. When I research this page by entering the URL into the "exact page" keyword research tool, Moz says it has no ranking keywords. How can a page be earning organic traffic without ranking for any keywords?
Moz Bar | | baystatemarketing0 -
Why do my Moz duplicate content results show me pages with no noticeably similar content?
Sometimes the "Pages with Duplicate Content" results under Content Issues show pages that, from what I'm able to see or otherwise test, have no duplicate content, save for the same navigation that exists on all of my pages. For example, a recent issue said that the following pages had duplicate content:
Moz Bar | | rickmic
https://freezerworks.com/index.php/html/slider-overlay
https://freezerworks.com/index.php/ufaqs/what-do-i-get-with-my-purchase-of-freezerworks
https://freezerworks.com/index.php/videos/fda-and-freezerworks-2
https://freezerworks.com/index.php/lims-testing-module Even a side-by-side of the page source in a text comparison tool shows nothing but navigation and scripts used in every page. Am I not seeing something?2 -
Utilizing the Moz Location in keywords vs including the location in the actual keyword phrase
I've searched the Moz blog as well as Q&A forums - but I specifically have a question about utilizing the Moz Location in keywords vs including the location in the actual keyword phrase. I've done both and get different rankings. For instance, I have "it company" with the Location set to Corpus Christi, Texas and our rank is #3. Then I also have "it company corpus christi" with the Location set to Corpus Christi, Texas and our rank is #6. Additionally, I threw in "it company corpus christi" with the Location set to National and our rank is #7. What is the best practice and what are the determining factors for the differences? Thank you in advance!
Moz Bar | | LBoxerger0 -
Page title not being read
Hi all I have been testing my site with the Moz on page grader recently and every page seems to come up with the same issues. The keyword doesn't seem to be in the title tag or the title tag isn't there. Could anyone tell me why this is? This is an example of a page it is happening on https://www.sbat.com/tips/todays-free-horse-racing-tips I used several versions of the Keyword including free horse Racing tips Horse racing tips and it seemed none of these were picked up.
Moz Bar | | gillies88880 -
My H1 is not readable in MOZ
I have tried a lot to no success. I can read the H1 in the source of the browser. And I have tried without any classes(joomla). Does anybody experienced something similar or have a clue why MOZ can't read my H1? Here is the link to the page: http://generalvarmepumpe.no/nordisk-luft-til-luft-varmepumpe/luft-til-luft-varmepumpe-tilbud-vinterkongen-2.html Appreciate any kind of help:)
Moz Bar | | zionray0 -
Duplicate Home Page in Moz Report
I recently ran a crawl report in MOZ for www.laddersfree.co.uk and it has picked duplicate pages for my home page. The two pages are www.laddersfree.co.uk and www.laddersfree.co.uk The CMS System my developer is using is Concrete5. If you look at the source code there is a Canonical <link rel="<a class="attribute-value">canonical</a>" href="http://www.laddersfree.co.uk/" /> I am very confused as to why there are two pages picked up in the crawl. Can someone offer any advice please? Thank you. Jason
Moz Bar | | gymmad0 -
Crawl Test cannot be seen on my PC. Using Windows 8.
I received and downloaded my Crawl Test. When I try to open it, my pc says "This app can't run on your PC. To find a version for your PC, check with the software publisher". I'm running Windows 8. Can I view my Crawl Test with my PC? Is there a work-around for this issue? Update I can apparently open my Crawl Test and view it as an Excel Spreadsheet. But when I download it and choose Save As, it saves it as a MS-DOS Application. This is my very first Crawl Test and I am not sure if I am doing everything right.
Moz Bar | | jameskoby010 -
Moz crawl sees meta description but there are none
I have a new site I have run a starter crawl on. The crawl came back saying some of the pages do indeed have meta descriptions. When I go to the same page and use the Moz Chrome toolbar it says they do not have meta descriptions. I also know they do not have meta descriptions. Are there any instances in which the Moz crawler would see them when they are not there?
Moz Bar | | SBXMedia0