Crawl test tool from SEOmoz - which URLs does it actually crawl?
-
I am using for the first time the crawl test tool from SEOmoz and I do not really understand which URLs the tool is going to crawl.
First, it says "enter any subdomain" --> why can´t I do the crawl for the root domain?
Second it says "we'll crawl up to 3,000 linked-to pages" --> does that mean that the tool crawls all internal links that it can find on the given domain?
Thanks for your help!
-
Hi Elke,
An initial Pro campaign will take up to 7 days to complete and will crawl up to a maximum 10,000 URLs for the root domain or sub domain that you choose to set it up for.
The Crawl Test tool is intended to provide a "quick start " option, which will report against the same diagnostic information, but for a limited number of URLs (3,000 & max of 2 domains per day). This allows you to get started on tidying up site issues while waiting for the full crawl to complete.
Since the Crawl Test tool allows you to run the quick crawl for any subdomain, it can also be useful for taking a look at the health of established sites you may be quoting on etc.
Hope that helps,
Sha
-
Alan, thank you for the answer. I am a Pro Member (trial version). But the only crawl diagnostics that I can find inside my campaign is a short Crawl Diagnostics summary which doesn´t include all the information that the crawl test tool gave me... I have just set up a campaign for the root domain, let´s see if the crawl diagnostics give me more information for that one.
-
When you become a pro meneber you can set up compaigns, then you can set up by root domain and crawl all sub domains.
the free version will crawl only one sudomain, such as
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Strange "?offset" URL found with content crawl issues
I recently recieved a slew of content crawl issues via Moz for URL's that I have never seen before For example:
Moz Pro | | HannahPalamara
Standard URL: https://skilldirector.com/news,
Newly identified URL: https://skilldirector.com/news?offset=1469542207800&category=Competency+Management). Does anyone know where the URL comes from and how to fix it?0 -
Facebook URLs, Anchor Text
I have a client that is considering a facebook url change. For ease of explanation, let's say their currently existing URL is facebook.com/Company123. I've googled their currently existing facebook url and found a dozen or so websites that include the text, "facebook.com/Company123". But, these results don't include websites that have an anchor text of, for example, "Facebook" and a link pointing to facebook.com/Company123. Has anybody had success tracking down any/all websites that point to a specific Facebook url? I've tried Open Site Explorer, OpenLinkprofiler, RankSignals, and SEO SpyGlass to no avail. Thank you!
Moz Pro | | OMTAnno0 -
Functionality of SEOmoz crawl page reports
I am trying to find a way to ask SEOmoz staff to answer this question because I think it is a functionality question so I checked SEOmoz pro resources. I also have had no responses in the Forum too it either. So here it is again. Thanks much for your consideration! Is it possible to configure the SEOMoz Rogerbot error-finding bot (that make the crawl diagnostic reports) to obey the instructions in the individual page headers and http://client.com/robots.txt file? For example, there is a page at http://truthbook.com/quotes/index.cfm month=5&day=14&year=2007 that has – in the header -
Moz Pro | | jimmyzig
<meta name="robots" content="noindex"> </meta name="robots" content="noindex"> This page is themed Quote of the Day page and is duplicated twice intentionally at http://truthbook.com/quotes/index.cfm?month=5&day=14&year=2004 and also at http://truthbook.com/quotes/index.cfm?month=5&day=14&year=2010 but they all have <meta name="robots" content="noindex"> in them. So Google should not see them as duplicates right. Google does not in Webmaster Tools.</meta name="robots" content="noindex"> So it should not be counted 3 times? But it seems to be? How do we gen a report of the actual pages shown in the report as dups so we can check? We do not believe Google sees it as a duplicate page but Roger appears too. Similarly, one can use http://truthbook.com/contemplative_prayer/ , here also the http://truthbook.com/robots.txt tells Google to stay clear. Yet we are showing thousands of dup. page content errors when Google Webmaster tools as shown only a few hundred configured as described. Anyone? Jim0 -
Using seomoz.
Hi, I'm scouring the web for new links, and am taking a closer look at OSE; I'm trying to gauge whether the PA & DA rating(s) in the results page for a domain search takes into account any penalty signals the site may be showing, and whether there is a 'real' way to identify how much value links from the source will give you (if for example, there's already x10 other links away from that page or x100 links away from that domain). Basically - what else needs to be taken into account when using this app to identify link sources, etc. Thanks, David
Moz Pro | | newstd1000 -
Crawl diagnostics taking too long
I started a crawl 2 days ago and it was still going after almost 48 hours so I deleted the entire campaign and resubmitted it. It's been 13 hours and still going. What happened to getting initial results in 2 hours? I've never had this problem and have run several campaign crawls here. Just wondering if there is a known issue I just can't seem to find? Thank you
Moz Pro | | LisaS130 -
How to Stop SEOMOZ from Crawling a Sub-domain without redoing the whole campaign?
I am using SEOMOZ for a client to track their website's performance and fix any errors and issues. A few weeks ago, they created a sub-domain (sub.example.com) to create a niche website for some of their specialized content. However, when SEOMOZ re-crawled the main domain (example.com), it also reported the errors for the subdomain. Is there any way to stop SEOMOZ from crawling the subdomain and only crawl the main domain? I know that can be done by starting a new campaign, but is there any way to work around an existing campaign? I'm asking because we would like to avoid the setting up the campaign again and losing the historical data as well. Any input would be greatly appreciated. Thanks!
Moz Pro | | TheNorthernOffice790 -
On page optimisation tool issues
When viewing my campaign and looking at the on page optimisation tool, I have a few issues. I seems to only shows the keywords I want rankings for and how optimised my homepage is for those keywords. Is there any way I can get it to analyse permanently specifc keywords for specific pages because my homepage isnt optimised for some keywords which are on my list, which I have optimised other pages for, and because its looking at my homepage its getting a really low grade, and looks really bad and frustrates me because I cant work this out. Any help greatly appreciated.
Moz Pro | | CompleteOffice1 -
SEOmoz Toolbar vs. Opensiteexplorer
Dumb question, why is the SEOmoz Toolbar reporting vastly different data than opensitexplorer? I had assumed they pulled from the same data set. False assumption? Am I misinterpreting the metrics? The discrepancies with which I am most confused are differences in number of root linking domains between OSE and Toolbar. Please enlighten me.
Moz Pro | | Gyi0