Crawl Diagnostics: How many pages (deep) will it crawl for dup content
-
Does anyone know how deep the crawl diagnostics will crawl when searching for dup content? Will it crawl the entire site, or will it only crawl "x" amount of pages?
Thanks!
-
Hello!
The standard and medium plans will each have a set limit to crawl up to 50,000 pages. The higher plans have adjustable limits, 10,000, 20,000, etc.
Hope this helps
-
The number of pages crawled depends on which plan you have - https://moz.com/products/pricing
- Standard = 250k
- Medium = 500k
- Large/Premium = 1.25m
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Crawl Report Increase in Errors?
Has anyone else noticed a huge increase over the past couple weeks in crawl issues in their dashboards? Without being able to see historical data week over week, I can't tell what's been added. Is this some update with the tool? I'm not seeing any health issues with this feature on the Moz Health page, it just seems strange that I'm seeing this across all our accounts.
Moz Bar | | WWWSEO0 -
Moz Content --- for SEO or simply user engagement?
What is the primary function of Moz Content? It looks like it is most useful for managing content as a user engagement tool. Our content strategy is centered on boosting organic placement - with user engagement as a nice but unessential side product. Besides providing general descriptive details of a site's content / authorship - how can Moz Content help with SEO?
Moz Bar | | cvonhassell0 -
How does a non-traditional TLD impact Moz's crawl test?
I have a client who moved from a .com to .academy domain 6 months ago, and their current crawl tests are coming back with a universal page authority of 1, along with 0 indexed backlinks. The previous version of the site had an average page authority of 35-40, the site architecture and content are nearly identical, and there are no other errors or red flags in the crawl report that would hold back their organic rankings. In fact, looking at the site's analytics account, I can see dozens of sites that provide current and properly functioning backlinks, non of which are listed on the crawl test. So the question is - is Moz currently unable to properly crawl a .academy (or any other non-traditional TLD) site, or is there some deeper issue with the site's SEO that I'm not seeing? Thanks!
Moz Bar | | ThinkAOR1 -
Www.site.com linking to pages www10.site.com
The root domain of the website in question is www.site.com but all subpages are on the subdomain www10.site.com (I'm pretty sure it's a subdomain, at least, used for load balancing?). A funny thing happens on this site with the moz toolbar. I visit a subpage, www10.site.com/articles/articletopic1 That page has a lot of links on it, all of them visibly going to the subdomain www10.site.com. However, the moz toolbar shows some of them as Internal links and most of them as External links. As far as I can tell, there is no real rhyme or reason to the difference between the links that are highlighted as Internal vs. External. The link structures vary greatly: Some are properly structured www10.site.com/blogs/category
Moz Bar | | Motava
And some are poor like www10.site.com/articles/show_articles.php?section=category1 So a couple questions here: Does this subdomain www10 have a detriment on the rankings of subpages?
What could possibly cause the internal links on these subpages to be highlighted as external pages with the moz toolbar?1 -
Problem Downloading Crawl Error Report PDF's
I am trying to download the PDF reports for the various 'crawl errors' - now some of them are quite large but would that justify why I am unable to download - the error is a straightforward one, see attached. Any ideas? Andy aDlViIN
Moz Bar | | TomKing0 -
Moz Crawler URL paramaters & duplicate content
Hi all, this is my first post on Moz Q&A 🙂 Questions: Does the Moz Crawler take into account rel="canonical" for search results pages with sorting / filtering URL parameters? How much time does it take for an issue to disappear from the issues list after it's been corrected? Does it come op in the next weekly report? I'm asking because the crawler is reporting 50k+ pages crawled, when in reality, this number should be closer to 1000. All pages with query parameters have the correct canonical tag pointing to the root URL, so I'm wondering whether I need to noindex the other pages for the crawler to report correct data?: Original (canonical URL): DOMAIN.COM/charters/search/mx/BS?search_location=cabo-san-lucas Filter active URL: DOMAIN.COM/charters/search/mx/BS?search_location=cabo-san-lucas&booking_date=&booking_days=1&booking_persons=1&priceFilter%5B%5D=0%2C500&includedPriceFilter%5B%5D=drinks-soft Also, if noindex is the only solution, will it impact the ranking of the pages involved? Note: Google and Bing are semi-successful in reporting index page count, each reporting around 2.5k result pages when using the site:DOMAIN.com query. The rel canonical tag was missing for a short period of time about 4 weeks ago, but since fixing the issue these pages still haven't been deindexed. Appreciate any suggestions regarding Moz Crawler & Google / Bing index count!
Moz Bar | | Vukan_Simic0 -
Site crawl errors - download list of all urls
Hi Ive provided my clients developers with the pdf reports of crawl errors but these seem to miss some urls I see there are lots of csv file download/email options Will the email csv button send a report of everything listing all urls that are missing from the pdfs ? if not will the more specific csv reports Would be good if i can press 1 button and get all issues listed with all urls It does look like this happens but i just want confirmed best way asap since need to provide reports urgently, any guidance much appreciated ? All Best Dan
Moz Bar | | Dan-Lawrence0 -
Rel Canonical and Moz Crawl
we have Rel Canonical tags set up on a few pages. When viewing the page source, the tags are correct. However, Moz Crawl results show the opposite. for example the page source, correctly shows, URL X with a Rel canonical Tag of URL Y
Moz Bar | | S.S.N
but.. Moz crawl is showing URL Y with a Rel Canonical Tag of URL X ..any thoughts why this would happen? which should i trust more?0