Is there a tool that works like crawl test that allows more than 3000 pages?
-
I enjoy using crawl test inside moz but I need to find a way to crawl all the pages on a site. It would probably be in the neighborhood of 10,000 urls. Does anyone know of a free tool and if not is there a paid tool that will do this?
-
Just a note that we've discontinued the old Crawl Test tool and have launched an entirely new On-Demand Crawl tool based on our upgraded Site Crawl engine (launched last year). The new tool has an enhanced UI, entirely rebuilt back-end, full export capability, and will save your old crawls for up to 90 days.
We've written up a sample case study or logged-in customers can go directly to On-Demand Crawl.
If you're looking to crawl more than 3,000 pages, you can also use campaign-based Site Crawl in Moz Pro.
-
Hi Brad!
Donna is spot-on; I'm especially a fan of Screaming Frog. That said, it looks like you're a Moz Pro subscriber, so you can use your Moz Analytics campaigns to crawl any site up to 50,000 pages.
Whenever you set up a new campaign, it automatically crawls the site it's set to track. It then updates every week, with the crawl results surfaced in Crawl Diagnostics. It looks like your account can hold up to 5 active campaigns, but they can always be deleted and more can be added.
Does that help at all?
-
Hi Brad,
Yes there are a couple. The two that come to mind are Screaming Frog and Xenu. Screaming frog is very popular among SEOs. The free version of will allow you to crawl up to 500 pages. A licensed version will allow you to crawl 10,000 pages or more and costs 99 pounds / year. (It's an Oxfordshire, UK company.) Highly recommended.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl-test not doesn't finish
Hello, I have used this crawl-test on 2 website 3 days ago, and it hasn't finished yet. I'm wondering if the crawler is on an infinite loop, or has crashed without sending back an error. I could re-launch the test, but if it's really still crawling, I don't want to loose any work in progress. Is there any way to check the status of a crawl?
Moz Bar | | Nobody16116145880332 -
Problem Downloading Crawl Error Report PDF's
I am trying to download the PDF reports for the various 'crawl errors' - now some of them are quite large but would that justify why I am unable to download - the error is a straightforward one, see attached. Any ideas? Andy aDlViIN
Moz Bar | | TomKing0 -
Multiple Page Title Elements
I have searched high and low for the source of this but just cannot find it. On-page grader gives the multiple page title elements warning, but I just can't identify where this is coming from. Can anyone, please, point me to the right direction?
Moz Bar | | Roselawn0 -
Confusing Moz Crawl?
Hi there, I am not sure if I am missing on something but the moz crawls are rather confusing. After singing in I have received 11 emails with crawls and today I have received again new, When I go to check there to the dashboard it shows 26 pages with issues. When I scroll down I see the pages with issue. Then when I click on the first page listed, to view the issues it says this: Rel Canonical
Moz Bar | | Rebeca1
Using rel=canonical suggests to search engines which URL should be seen as canonical. For this site: http://villasdiani.com/ but we have sorted out the canonical issues a long time ago. Is this a wrong information or is it really true that we do not specify the canonical for our site? Then the second page with issue is there listed http://villasdiani.com/beach-villas/ and it says: Duplicate Page Title
You should use unique titles for your different pages to ensure that they describe each page uniquely and don't compete with each other for keyword relevance. But it does not point out which page is duplicate with this one! I do not have any other page named the same way. It also says in Issues overview 26pages with issues, but it shows on the bottom only 5 under and when I click on view more it brings me to high priority issues where is 0. The most is freaking me out this report: When I click on links, there are listed on the bottom the pages with highest authority among which I found this http://villasdiani.com/db I have never created this kind of page! Funny enough when I click on it it really open that page! How this can be??? In issues overview it also shows on the bottom, right corner 11 page with duplicate content but when I click on it to review it it brings me to high priority issues windows where is not displayed anything Can somebody advice me regarding of this. I have sign up here to learn and sort out the problems with the site but so far I am only getting more confused here. Thank you very much for looking into this.0 -
Moz reporting appropriate Canonical tag usage but no canonical tag on page !?
I take it this means that the page in question has been referenced via a different pages canonical tag but that the page in question itself does not have a self referencing canonical tag (and that it should do) cheers dan
Moz Bar | | Dan-Lawrence0 -
Inconsistent On-Page Grader results
Hello everyone! I am fairly new to the MOZ tools but I'm finding some inconsistencies in the On-Page grader.. I have tested several pages (some of which I know have doos and bad SEO) and the results I keep getting are very volatile. For example, I have looked at testing http://www.sportingwebdesign.com for "Sports Website Design" - one minute it gave me a grade F, then then 2 minutes after I refresh the page and it gave me a grade A.. I am very confused and a bit worried that this tool may be a bit volatile... Anyone have any thoughts?
Moz Bar | | Social_Media_Mentors0 -
Is there a way to do a mass lookup of Page Authority?
Meaning, is there a way I could look at the Page Authority for 50 pages all in one place? Or do I need to use the MozBar and go page by page? Just looking to save some time. Thanks.
Moz Bar | | JoshKimber0