Settings to crawl entire site
-
Not sure what happened but I started a third campaign yesterday and only 1 pages was crawled,
The other two campaigns has 472 and 10K respectively.
What is the proper setting to choose in the beginning of campaign setup to have the entire site crawled. Not sure what I did different and I must be reading the instructions incorrectly.
Thanks,
Don
-
The most common reasons for this are discussed at https://seomoz.zendesk.com/entries/409821-why-isn-t-my-site-being-crawled-you-re-not-crawling-all-my-pages. If those don't solve your problem, email help@seomoz.org and we'll be able to sort it out at the help desk.
Thanks!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Crawl Test error
Moz crawl test show blank report for my website test - guitarcontrol.com. Why??? Please suggest.
Moz Pro | | zoe.wilson170 -
How to Properly Set Up My Campaign
My url www.pcf.org redirects to ttp://www.pcf.org/site/c.leJRIROrEpH/b.5699537/k.BEF4/Home.htm because of my old CMS and this cannot be changed. Should I set it up as a Subdomain, root domain, or folder? For example using the homepage url http://www.pcf.org/site/c.leJRIROrEpH/b.5699537/k.BEF4/Home.htm I get the following results: subdomain: You've decided to set up a subdomain campaign, but entered the subfolder path:
Moz Pro | | mstanwyck
www.pcf.org/site/c.leJRIROrEpH/b.5699537/k.BEF4. Don't worry, we'll switch that for you and crawl everything on your subdomain: www.pcf.org. If you meant to set this up to only crawl pages in the subfolder /site/c.leJRIROrEpH/b.5699537/k.BEF4, click "Go back and Change" and choose the subfolder option in step 1. root domain: You've decided to set up a root domain campaign, but entered the subfolder path: www.pcf.org/site/c.leJRIROrEpH/b.5699537/k.BEF4. Don't worry, we'll switch that for you and crawl everything in the subfolder /site/c.leJRIROrEpH/b.5699537/k.BEF4. subfolder: It looks like you submitted a URL that relates to an individual file: www.pcf.org/site/c.leJRIROrEpH/b.5699537/k.BEF4/Home.htm
rather than a folder:
www.pcf.org/site/c.leJRIROrEpH/b.5699537/k.BEF4. Don’t worry, we’ll switch that for you and crawl everything in that folder instead. If you meant to set this up for a different folder, click ”Go back and change” and enter a different path. To complicate things further, as you move through the site, the only part of the URL that starys consistent 100% of the time is: http://www.pcf.org/site/c.leJRIROrEpH/ Depending on where you are on the site, it appends the url. For example: One page is http://www.pcf.org/site/c.leJRIROrEpH/b.6185669/k.9792/Sherry_Galloway.htm while another is http://www.pcf.org/site/c.leJRIROrEpH/b.5800843/k.94DE/Everyday_Heroes.htm0 -
Not all pages are being crawled
I am set up on the PRO plan, I was under the impression that it would crawl up to 10,000 pages. My site has just over 200 pages, but whenever I am crawled it only crawls 121 pages. Is this normal? It's hard to know how reliable my data is because a significant amount of pages are missing.
Moz Pro | | KristinHarding0 -
Crawl Diagnostics - unexpected results
I received my first Crawl Diagnostics report last night on my dynamic ecommerce site. It showed errors on generated URLs which simply are not produced anywhere when running on my live site. Only when running on my local development server. It appears that the Crawler doesn't think that it's running on the live site. For example http://www.nordichouse.co.uk/candlestick-centrepiece-p-1140.html will go to a Product Not Found page, and therefore Duplicate Content errors are produced. Running http://www.nhlocal.co.uk/candlestick-centrepiece-p-1140.html produces the correct product page and not a Product Not Found page Any thoughts?
Moz Pro | | nordichouse0 -
Crawl Diagnostics Update
I have corrected some errors in my SEOMoz Crawl Diagnostics, however the errors are still showing. It says a crawl has happen since. Any idea's why?
Moz Pro | | petewinter0 -
Crawl slow again
Once again the weekly crawl on my site is very slow. I have around 441 pages in the crawl and this has been running for over 12 hours. This last happened two weeks ago (ran for over 48 hours). Last week's crawl was much quicker (not sure exactly how long but guessing an hour or so). Is this a known issue and is there anything that can be done to unblock it? Weekends are the best time for me to assess and respond to changes I have made to my site so having this (small) crawl take most of the weekend is really quite problematic. Thanks. Mark
Moz Pro | | MarkWill0 -
Do crawl reports see canonical tags?
Greetings, I just redesigned my site, www.funderstanding.com, and have the old site pointing to the new site via canonical URLs. I had a new crawl test run and it showed a large amount of duplicate content. Does the SEO Moz crawl tool validate canonical urls and adjusts the duplicate content count or is this note considered? FYI, I sent from no duplicate content to having 865 errors since the redesign went up so that seems suspicious. I would think though that assuming the canonical tag were used properly, and I hope it is?, that this would not be a problem?? All help with this is most appreciated. Eric
Moz Pro | | Ericc220