How does SEOMoz crawl sites? Does it follow the sitemap?
-
I only ask because in my report it returned a few "no follow" links. Was wondering how SEOMoz operates.
-
Rogerbot, our crawler, obeys the robots.txt nofollow rules associated with your site. For some reason your robots.txt file is blocking our crawler from seeing those pages. For more information on how the tool works, you can view this incredible resource: http://www.seomoz.org/help/rogerbot-crawler that explains how our crawler works in vivid detail. I hope this helps!!
-
My guess is they spider the way Googlebot spiders - finding links on your home page, following those links, and finding more links on the subsequent pages, so on and so forth. I doubt they look at the sitemap, since it rarely reflects the true importance of pages on your site, which tend to be linked to more often.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it me or is SEOMOZ Q&A slow
I try and answer many questions here on SEOMoz Q&A, but the page load speed seems so slow I think my browser is going to time out. I don't remember it being this slow. Anybody else noticing slow loading (especially when asking, answering, or replying to questions)?
Moz Pro | | Francisco_Meza1 -
How can I see the URL's affected in Seomoz Crawl when Notices increase
Hi, When Seomoz crawled my site, my notices increased by 255. How can I only these affected urls ? thanks Sarah
Moz Pro | | SarahCollins0 -
Crawl Diagnostics - unexpected results
I received my first Crawl Diagnostics report last night on my dynamic ecommerce site. It showed errors on generated URLs which simply are not produced anywhere when running on my live site. Only when running on my local development server. It appears that the Crawler doesn't think that it's running on the live site. For example http://www.nordichouse.co.uk/candlestick-centrepiece-p-1140.html will go to a Product Not Found page, and therefore Duplicate Content errors are produced. Running http://www.nhlocal.co.uk/candlestick-centrepiece-p-1140.html produces the correct product page and not a Product Not Found page Any thoughts?
Moz Pro | | nordichouse0 -
Help with URL parameters in the SEOmoz crawl diagnostics Error report
The crawl diagnostics error report is showing tons of duplicate page titles for my pages that have filtering parameters. These parameters are blocked inside Google and Bing webmaster tools. I do I block them within the SEOmoz crawl diagnostics report?
Moz Pro | | SunshineNYC0 -
SEOmoz showing crawl errors but webmastertools says no errors, need help!
Hi this is my first question and i couldnt find a similar question on here. basically i have a clients website that is showing 150 duplicate page titles and content errors plus others. SEOmoz analysis is showing me for example is 3 duplicate hompage URLS: 1.www.domain.com 2.domain.com 3.www.domain.com/index.html all 3 are the same page. after explaining to the guy (who built the website) the errors, he ensured me that the main URL is URl 1. and the other 2 are 301 redirects. however SEOmoz analysis doesnt seem to change the results and webmastertools doesnt seem to show any errors at all. also if i try all 3 URL's there are no redirects to URL 1. any help or clarity would be awesome! Thanks e-bob
Moz Pro | | bobsnowzell0 -
My Campaign has been crawling for about a week now
Can anyone tell me why one of my campaigns has been stuck in crawl mode for about a full week and it is still not done?!?!
Moz Pro | | nazmiyal0 -
Too many pages indexed in SEOMoz
I am running a campaign for a client that has 86 pages via Google and SEmoz is up to almost 10K pages. I am really confused. Any ideas?
Moz Pro | | LaurieK130