How does SEOMoz crawl sites? Does it follow the sitemap?
-
I only ask because in my report it returned a few "no follow" links. Was wondering how SEOMoz operates.
-
Rogerbot, our crawler, obeys the robots.txt nofollow rules associated with your site. For some reason your robots.txt file is blocking our crawler from seeing those pages. For more information on how the tool works, you can view this incredible resource: http://www.seomoz.org/help/rogerbot-crawler that explains how our crawler works in vivid detail. I hope this helps!!
-
My guess is they spider the way Googlebot spiders - finding links on your home page, following those links, and finding more links on the subsequent pages, so on and so forth. I doubt they look at the sitemap, since it rarely reflects the true importance of pages on your site, which tend to be linked to more often.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Open Site Explorer detects links from a site that redirects to it? How is this possible?
I was checking external links to a site: scte-brasilien.de/ and was wondering why there are so many links pointing from another domain (reisen-nach-brasilien.com/) although it redirects to scte-brasilien.de/? So when checking the redirecting domain OSE knows its redirecting... The URL you've entered redirects to another URL. We're showing results for scte-brasilien.de/ since it is likely to have more accurate link metrics. See data forreisen-nach-brasilien.com/ instead? How is this possible? best regards
Moz Pro | | inlinear
Holger0 -
SEO on-demand crawl
what happened to the on-demand crawl you could do in PRO when they switched to the new MOZ site?
Moz Pro | | Vertz-Marketing0 -
SEOMOZ Support: Domain Name Change in SEOMOZ
My client is changing their domain name. They are redirecting (301) all of their URLs to the new domain. Do I need to do anything in SEOMOZ to preserve the campaign?
Moz Pro | | DynoSaur0 -
SEOmoz Campaign Setup: What would you do?
I've been using moz for nearly a year now. Initially when I set up our site (rac.com.au) it was as a single campaign for the rac.com.au domain. However, our business has a number of different product offerings and marketing teams (Insurance, finance, travel/tourism, automotive, security etc) - would I be better off setting each of these 'sub' brands as a campaign on their own using the 'folder' campaign setup? For example, have a campaign setup for the Insurance business (rac.com.au/insurance) folder, Finance business (rac.com.au/finance) and so forth? The advantage I see to the folder campaign setup is more specific reporting for each of the business units. Are there disadvantages to this? Any recommendations would be greatly appreciated. Cheers Ryan
Moz Pro | | Hutch_e0 -
Only few pages (308 pages of 1000 something pages) have been crawled and diagnosed in 4 days, how many days till the entire website is crawled complete?
Setup campaign about 4-5 days ago and yesterday rogerbot said 308 pages were crawled and the diagnostics were provided. This website has over 1000+ pages and would like to know how long it would take for roger to crawl the entire website and provide diagnostics. Thanks!
Moz Pro | | TejaswiNaidu0 -
How do you tell SEOmoz to ignore a subdomain?
We have a subdomain that I don't want to show up in our root SEOmoz campaign. How do I tell SEOmoz to ignore it?
Moz Pro | | wcsjohn0 -
How long should the weekly crawl take
Mine started yesterday afternoon and it's now almost 11pm on Sunday. 30+ hours and still not finished (and no progress indicator). 438 pages quoted as being crawled. That's not normal - right? I have made a bunch of changes based on last weeks crawl so I have been eagerly waiting for this to finish But 30 hours?.... Thanks. Mark
Moz Pro | | MarkWill0 -
Only 1 page is being crawled by SEOmoz for the last 2 crawls
I would like to ask for the possible problem plus solution on one of our campaigns. Only 1 page is being crawled by SEOmoz for the last 2 crawls. Before the last two crawls, SEOmoz crawls numerous pages and we can’t think of a possible reason for this error. For this particular campaign , there are no data --- no errors, warnings and notices. Thanks!
Moz Pro | | TheNorthernOffice790