Crawl test limitaton - ways to take advantage of large sites?
-
Hello
I have a large site (120,000+) and crawl test is limited to 3,000 pages.
I want to know if you have a way to take advantage to crawl a type of this sites. Can i do a regular expression for example?
Thanks!
-
Hi there. Kristina from Moz's Help Team here.
The Crawl Test tool is limited to only 3,000 pages by design as the full Site Crawl that happens within your campaign(s) is a much larger, weekly crawl that occurs.
All Moz Pro Standard & Medium subscription levels (you currently have access to a Standard account) default all Site Crawls within a Campaign at a max of 50,000 pages. If you're interested in crawling more than 50,000 pages within a single Campaign, it would require subscribing to a higher subscription.
You can view all of our subscription plans on our pricing page here: https://moz.com/products/pro/pricing
As always, you can reach out to our team in the future with product questions such as this one by either emailing help@moz.com or clicking on the blue chat box on the lower right hand corner of your screen while in the product itself.
I hope this helps but please let me know if there's anything else I can assist with!
Thank you,
-Kristina
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Server blocking crawl bot due to DOS protection and MOZ Help team not responding
First of all has anyone else not received a response from the help team, ive sent 4 emails the oldest one is a month old, and one of our most used features on moz on demand crawl to find broken links doesnt work and its really frustrating to not get a response, when we're paying so much a month for a feature that doesnt work. Ok rant over now onto the actual issue, on our crawls we're just getting 429 errors because our server has a DOS protection and is blocking MOZ's robot, im sure it will be as easy as whitelisting the robots IP, but i cant get a response from MOZ with the IP. Cheers, Fergus
Feature Requests | | JamesDavison0 -
Why my site not crawl?
my error in dashboard: **Moz was unable to crawl your site on Jul 23, 2020. **Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster i thing edit robots.txt how fix that?
Feature Requests | | alixxf0 -
Any way to programatically retrieve a campaign's Search Visibility score?
I would like to retrieve a campaign's Search Visibility score, and preferably historical too, in an automated fashion. However, as far as I know the API has no access to this. Is there something I'm missing? If not, is there a formula I can use to calculate this score?
Feature Requests | | MaddenMediaSEO1 -
Moz crawler is not able to crawl my website
Hello All, I'm facing an issue with the MOZ Crawler. Every time it crawls my website , there will be an error message saying " **Moz was unable to crawl your site on Sep 13, 2017. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. " We changed the robots.txt file and checked it . but still the issue is not resolved. URL : https://www.khadination.shop/robots.txt Do let me know what went wrong and wjhat needs to be done. Any suggestion is appreciated. Thank you.
Feature Requests | | Harini.M0 -
MOZ Site Crawl - Ignore functionality question
Quick question about the ignore feature found in the MOZ Site Crawl. We've made some changes to pages containing errors found by the MOZ Site Crawl. These changes should have resolved issues but we're not sure about the "Ignore" feature and do not want to use it without first understanding what will happen when using it. Will it clear the item from the current list until the next Site Crawl takes place. If Roger finds the issue again, it will relist the error? Will it clear the item from the list permanently, regardless if it has not been properly corrected?
Feature Requests | | StickyLife1 -
Is there a way to schedule automatic weekly .csv reports for the Tracked Keywords Overview?
Using The Custom Reports tool, I only managed to get PDF reports. It would be useful to automatically receive .csv reports by email. Any idea how?
Feature Requests | | Digital-Sun0 -
Way to track public's use of keywords over time in MOZ?
I apologize for the basic question but is there a way to track within MOZ the public's use of my keywords over time? My traffic understandably goes down certain times of the year -- holidays, vacation season, back to school, etc. What I want to see is whether my share of the traffic available is staying steady, slipping or even increasing. Sometimes I'll see a big spike in traffic that I suspect is related more to a general increase in traffic in my keywords than anything else. Thanks for any help.
Feature Requests | | NCCompLawyer0 -
Does Rogerbot support SNI? - And if not All sites behind Cloudflare are probably uncrawlable.
Hi All, Recently i've noticed that Moz can no longer crawl any of my wbsites that are behind cloudflare. After speaking to CF support, we have come to the conclusion that a recent change by CF to force SNI for all SSL domains is the issue. As this is likely to affect almost everyone using Cloudflare could anyone please confirm this? In the mean time, one of the websites currently blocked is on CF enterprise, so we've requested SNI be turned off. Will test crawl again when thats done to confirm SNI is the issue.
Feature Requests | | N1ghteyes5