Can SEO Moz perform a full site crawl and provide a report showing all URLs within an existing domain?
-
We are conducting a site redesign and need to get an idea of all pages that are out there on our domain (in some report fashion). This would help for discovery and cleanup as we re-work the site and move to a new CMS.
Thanks
-
Hey Stephen,
THanks - I have heard of Xenu many times before. I'll try that out.
I appreciate the input!
-
just going to throw in an alternative for this type of thing Xenu Link sleuth - free, easy and powerful.
Yes that site looks like ass but the tool is great. Dr Pete and others have done "how to use" posts on seomoz - just search xenu on seomoz
-
Yes - that's exactly it. Can I just do an export of the campaign crawl? or is that found somewhere else? Forgive my ignorance...still new to the tool and just can't seem to find.
-
Are you looking at your crawl report? It is available in an Excel file which provides the exact information you desire.
-
Hey Ryan,
Thanks for the response.
It's not a huge site. I would say there's probably around 600 total pages. It definitely looks like the SEO crawler has crawled the entire site, but I can't seem to find a way to just get a one-shot view of all the pages it's accessed.
Any ideas on where this can be found?
Thanks again!
-
How many pages are on your site?
The SEOmoz crawler will do exactly what you want but there is a URL limit. You should also understand it will experience the same limitations of any crawler. It wont list pages which behind a user log-in nor pages which are blocked by robots.txt. I am pretty sure the SEOmoz crawler doesn't list secure pages either but last I heard they were working on including them.
If your site has any island pages they wont be discovered either.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does Google count the domain name in its 115-character "ideal" URL length?
I've been following various threads having to do with URL length and Google's happiness therewith and have yet to find an answer to the question posed in the title. Some answers and discussions have come close, but none I've found have addressed this with any specificity. Here are four hypothetical URLs of varying lengths and configurations: EXAMPLE ONE:
Web Design | | RScime25
my-big-widgets-are-the-best-widgets-in-the-world-and-come-in-many-vibrant-and-unique-colors-and-configurations.html (115 characters) EXAMPLE TWO: sample.com/my-big-widgets-are-the-best-widgets-in-the-world-and-come-in-many-vibrant-and-unique-colors-and-configurations.html (126 characters) EXAMPLE THREE: www.sample.com/my-big-widgets-are-the-best-widgets-in-the-world-and-come-in-many-vibrant-and-unique-colors-and-configurations.html (130 characters) EXAMPLE FOUR: http://www.sample.com/my-big-widgets-are-the-best-widgets-in-the-world-and-come-in-many-vibrant-and-unique-colors-and-configurations.html (137 characters) Assuming the examples contain appropriate keywords and are linked to appropriate anchor text (etc.,) how would Google look upon each? All I've been able to garner thus far is that URLs should be as short as possible while still containing and contextualizing keywords. I have 500+ URLs to review for the company I work for and could use some guidance; yes, I know I should test, but testing is problematical to the extreme; I look to the collective/accumulated wisdom of the MOZVerse for help. Thanks.1 -
How can we improve our e-commerce site architecture to help best preserve Page Authority?
Today I installed the SEOMoz toolbar for Firefox (very cool, highly recommended). I was comparing our site http://www.ccisolutions.com to this competitor: http://www.uniquesquared.com For the most part, the deeper I go in our site the more the page authority drops. We have a few exceptions where the page authority of a subcategory page is actually better than the cat. page one level up. In comparison, when I was looking at http://www.uniquesquared.com I noticed that their page authority stays at "21" on every single category page I visit. Are you seeing what I'm seeing? Is this potentially a problem with the tool bar or, is there something significantly different about their site architecture that allows them to maintain that PA across all category and sub category pages? Is there something fundamentally wrong with our (http://www.ccisolutions.com) site architecture? I understand that we have longer URLs, but this is an old store with a lot of SKUs, so we have decided not to remove the /category/ and /product/ from the URLs because the 301 redirects that would result wouldn't pass all of the authority they've built up over the years. Interested to know viewpoints on the site architecture and how it might be improved. Thanks!
Web Design | | danatanseo0 -
Image sliders & site speed
We are having a new website designed using WordPress and the Genesis framework. We wanted to include header image sliders on a number of internal site pages, but our designer says that sliders on more than just the home page will slow down the site significantly. How much could they slow down the site, and what can be done to minimize their effect on site speed?
Web Design | | GordyH0 -
SEO and Server Connectivity....
Good Morning/Evening Mozzers, I arrive at work this morning with 5 emails from GWT for my separate domains reading, **"Googlebot can't access your site - **Over the last 24 hours, Googlebot encountered 39 errors while attempting to connect to your site. Your site's overall connection failure rate is 15.1%." I have passed this on to the Web Dev team to resolve ASAP. My Question, will server connectivity issues harm my rankings? Is there a danger if this continues that URL's could be de-indexed? Input would be greatly appreciated.
Web Design | | RobertChapman0 -
Why can't I ask this question - It is not too short
I tried to post a question which was at least 15 words long and received an error saying the question was less than 5 characters QrXcp
Web Design | | FFTCOUK0 -
Does page speed worth for SEO?
I always broken my head to try to follow all pagespeed guidelines. I increase my pagespeed significantly, but i didnt saw any effect in my SEO performance. In my keywords, my concorrents are crap on it (I have score of 90 and they are at 60-70).Does google gives importance to it?
Web Design | | Naghirniac0 -
How can the Web site designer and the SEO strategist work together peacefully?
The organization I work for has decided to re-design or re-develop the existing company Web site. My part in this project is to come up with new features to add to the site, as well as making the site SEO-friendly (copywriting, link-building, keyword research, etc.). I don’t know a thing about Web site design, coding, format, etc., and I guess I will have to work with a designer on this project. How would I go about finding a Web site designer? Should they have some SEO knowledge? How much designer, coding and site structure knowledge should I have? And how do we not infringe on one another as we work together? (Sorry so many questions.)
Web Design | | Obie0 -
URL question for SEO...
I'm thinking of creating a new url off an existing url and was wondering if there would be any impact. For example I have the URL www.baseball.com and rather than secure a new url for a new product launch such as www.newbaseballproduct.com I want to do newproduct.baseball.com Will this hurt my SEO rankings for this new site? Basically wanting to figure out if this will hurt me or not? Should I get a new url or re-utilize an existing URL... really for a landing page/micro site, etc.,
Web Design | | gritacco0