Crawl Diagnostics: How many pages (deep) will it crawl for dup content
-
Does anyone know how deep the crawl diagnostics will crawl when searching for dup content? Will it crawl the entire site, or will it only crawl "x" amount of pages?
Thanks!
-
Hello!
The standard and medium plans will each have a set limit to crawl up to 50,000 pages. The higher plans have adjustable limits, 10,000, 20,000, etc.
Hope this helps
-
The number of pages crawled depends on which plan you have - https://moz.com/products/pricing
- Standard = 250k
- Medium = 500k
- Large/Premium = 1.25m
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl tests stuck in queue
I have tried to run a number of crawl tests recently for our client's sites outside the US and they have been stuck in the queue for over a week. 3 of them completed, but then 5 are stuck. Anyone experience this? I haven't seen anything about crawl tests having issues right now.
Moz Bar | | rmcgrath810 -
Site Crawl report show strange duplicate pages
Beginning in early in Feb, we got a big bump in duplicate pages. The URLs of the pages are very odd: Example URL:
Moz Bar | | Neo4j
http://firstname.lastname@website.com/dir/page.php
is duplicate with http://website.com/dir/page.php I checked though the site, nginx conf files, and referral pages, and could not find what is prefixing the pages with 'http://firstname.lastname@'. Any ideas? The person whose name is 'Firstname Lastname' is stumped as well. Thanks.0 -
On-Page Grader fails?
I have optimized the following page for the norwegian word "Dusjkabinett": https://www.vvskupp.no/Produkter/Baderom/Dusjkabinett The On-Page grader gives the URL an F-grade. BUT as you can see it seems like the tool fails in giving a correct report. See attachment. Can anybody help me understand? pCyEYaWd3
Moz Bar | | moggiew0 -
Crawl Test Takes Long Time
Hi Moz, I have submitted our website for a crawl test. Usually it would only take a few hours to do the crawl. However this time, it takes quite long time and the result still shows in progress 😞 This is a small website which only contains less than 10 pages. Just wondering if this is our website setting issue or it is a technical issue at your end? Many thanks in advance. sFjAERG.png
Moz Bar | | russellbrown0 -
I got a 404 in the Crawl Test Tool Report
I, yesterday i ran an crawl on http://www.everlastinggarden.nl and i get an 404. Does anybody know why this happens? <colgroup><col width="1535"></colgroup>
Moz Bar | | IMforYou
| # ---------------------------------------- |
| Crawl Test Tool Report | Moz,http://pro.seomoz.org/tools/crawl-test |
| www.everlastinggarden.nl |
| Report created: 15 Jul 18:34 |
| # ---------------------------------------- |
| URL,Time Crawled,Title Tag,Meta Description,HTTP Status Code,Referrer,Link Count,Content-Type Header,4XX (Client Error),5XX (Server Error),Title Missing or Empty,Duplicate Page Content,URLs with Duplicate Page Content (up to 5),Duplicate Page Title,URLs with Duplicate Title Tags (up to 5),Long URL,Overly-Dynamic URL,301 (Permanent Redirect),302 (Temporary Redirect),301/302 Target,Meta Refresh,Meta Refresh Target,Title Element Too Short,Title Element Too Long,Too Many On-Page Links,Missing Meta Description Tag,Search Engine blocked by robots.txt,Meta-robots Nofollow,Blocked by X-robots,X-Robots-Tag Header,Blocked by meta-robots,Meta Robots Tag,Rel Canonical,Rel-Canonical Target,Blocking All User Agents,Blocking Google,Blocking Yahoo,Blocking Bing,Internal Links,Linking Root Domains,External Links,Page Authority |
| http://www.everlastinggarden.nl,2014,404 : Received 404 (Not Found) error response for page.,Error attempting to request page | Best regards, Jos0 -
Can't delete items from the on page grader
I check every single box and they don't delete. This is driving me nuts. Please can you delete them for me because I am not impressed with this AT ALL. In fact I am getting so cross I am in danger of screaming hysterically which might get me the sack and it would be all your fault. That was slightly tongue in cheek, but please can you fix it please. please.
Moz Bar | | CommT0 -
Find all the back links to all the posts/pages within the blog subdirectory only.
Hi, I am new to Moz. I using the open site explorer to find backlinks for a website's blog. The website itself is huge. I want to find all the backlinks to all the posts/pages within the blog subdirectory only. Not the regular website. I ran a few reports, but it is giving me links to that page, not all the sub pages.Fi
Moz Bar | | DarrenD0 -
Is it no longer possible to export the On-Page grader reports?
Hi Everyone, I was just trying to export the On Page Grader report for my clients website, and found that there is no longer that option. Is this a permanent change with the new branding? Would appreciate any feedback. Thanks very much.
Moz Bar | | Peter.Huxley590