Can SEO Moz perform a full site crawl and provide a report showing all URLs within an existing domain?
-
We are conducting a site redesign and need to get an idea of all pages that are out there on our domain (in some report fashion). This would help for discovery and cleanup as we re-work the site and move to a new CMS.
Thanks
-
Hey Stephen,
THanks - I have heard of Xenu many times before. I'll try that out.
I appreciate the input!
-
just going to throw in an alternative for this type of thing Xenu Link sleuth - free, easy and powerful.
Yes that site looks like ass but the tool is great. Dr Pete and others have done "how to use" posts on seomoz - just search xenu on seomoz
-
Yes - that's exactly it. Can I just do an export of the campaign crawl? or is that found somewhere else? Forgive my ignorance...still new to the tool and just can't seem to find.
-
Are you looking at your crawl report? It is available in an Excel file which provides the exact information you desire.
-
Hey Ryan,
Thanks for the response.
It's not a huge site. I would say there's probably around 600 total pages. It definitely looks like the SEO crawler has crawled the entire site, but I can't seem to find a way to just get a one-shot view of all the pages it's accessed.
Any ideas on where this can be found?
Thanks again!
-
How many pages are on your site?
The SEOmoz crawler will do exactly what you want but there is a URL limit. You should also understand it will experience the same limitations of any crawler. It wont list pages which behind a user log-in nor pages which are blocked by robots.txt. I am pretty sure the SEOmoz crawler doesn't list secure pages either but last I heard they were working on including them.
If your site has any island pages they wont be discovered either.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
ECWID Ecommerce Sites. No Custom URLS?
Is there any way possible to be able to name product urls in website that use ECWID for their ecommerce? They have long and "dirty" urls. For example this running boards site: http://www.runningboards4less.com/general-motors#!/~/product/category=6593890&id=28043027 Isn't this hurting the overall SEO of the site? Especially product pages?
Web Design | | Atlanta-SMO0 -
Is there something fundamentally wrong with our site architecture?
Hi everyone! Could a few of you brilliant people take a look at the architecture of this site http://www.ccisolutions.com, and let me know if you see any obvious problems? I have run the site through XENU, and all of our most important pages, including categories and products, are no deeper than level 3. Everything deeper than that is, in most cases, an image, a pdf or an orphaned page (of which we have thousands). Could having thousands upon thousands of orphaned pages be having a more hurtful effect on our rankings than our site architecture? I have made loud noises and suggested that duplicate content, site speed and dilution of page authority due to all those orphaned pages are some of the primary reasons we don't rank as well as we could. But, I think those suggestions just aren't sexy or dramatic enough, so there is much shaking of heads and discussion that it must be something fundamentally wrong with site architecture. I know re-arranging the furniture is more fun than scrubbing the floors, but I think our problems are more about fundamental cleanup than moving things around What do you think?
Web Design | | danatanseo0 -
Is it better to redirect a url or set up a landing page for a new site?
Hi, One of our clients has got a new website but is still getting quite a lot of traffic to her old site which has a page authority of 30 on the home page and has about 20 external backlinks. It's on a different hosting package so a different C block but I was wondering if anyone could advise if it would be better to simply redirect this page to the new site or set up a landing page on this domain simply saying "Site has moved, you can now find us here..." sort of idea. Any advice would be much appreciated Thanks
Web Design | | Will_Craig0 -
I've set up my own site which is still fairly new but I'm a bit concerned that there is a bloackage SEO wise somewhere because when I try to crawl the site on SEOmoz it only crawls one page.
I'm really baffled and none of my research has shed much light on it. My url is www.emporiumofmanliness.co.uk I'd really appreciate any help! Thanks
Web Design | | JoshED0 -
How can we improve our e-commerce site architecture to help best preserve Page Authority?
Today I installed the SEOMoz toolbar for Firefox (very cool, highly recommended). I was comparing our site http://www.ccisolutions.com to this competitor: http://www.uniquesquared.com For the most part, the deeper I go in our site the more the page authority drops. We have a few exceptions where the page authority of a subcategory page is actually better than the cat. page one level up. In comparison, when I was looking at http://www.uniquesquared.com I noticed that their page authority stays at "21" on every single category page I visit. Are you seeing what I'm seeing? Is this potentially a problem with the tool bar or, is there something significantly different about their site architecture that allows them to maintain that PA across all category and sub category pages? Is there something fundamentally wrong with our (http://www.ccisolutions.com) site architecture? I understand that we have longer URLs, but this is an old store with a lot of SKUs, so we have decided not to remove the /category/ and /product/ from the URLs because the 301 redirects that would result wouldn't pass all of the authority they've built up over the years. Interested to know viewpoints on the site architecture and how it might be improved. Thanks!
Web Design | | danatanseo0 -
Branding site
We are starting a new used car site in latam, and we are in the process of developing the brand, part of the team would like to have the "auto" (spanish for car" in the name and some no. The question is, form a SEO point of view, how strong signal is to have the string auto as part as the URL, and can you have a complete unrelated url and have several URLs with the auto string on it pointing in a 301 way to the non unrelated brand name as a way around? (sorry about my english, I hope you can understand the question) PS, yes, there are a lot of queries involving auto in our target market.
Web Design | | Bligoo0 -
Mobile Sitemap for Site with Media Queries
I'm doing SEO for a site. It uses Media Queries and the CSS to automatically resize the site for the screen size in use. I.e. the site detects the screen size of say an iPhone and the CSS knows which elements to hide for that screen size and still make it look good. This is great because it will automatically cut down the content to display nicely on small screens - obviating the need for a separate mobile site. What kind of sitemap should be generated since the urls are for desktop and mobile use? Yoast (sweet SEO) said it should have both regular and mobile style sitemap to get both the regular and mobile bots to visit, but didn't elaborate on how that sitemap should look. Do you have a recommendation for how exactly the sitemap should look? Should the sitemap have the urls all twice, i.e. once regular and once with the mobile indicator?
Web Design | | GregoryHaze1 -
Which ecommerce platform is best for SEO?
We currently run our eyewear store on osCommerce. However, for various reasons we are considering a redevelopment onto another platform, the most obvious choice being Magento. What might the advantage and improvement in SEO with such a change and is the pain worth the gain???
Web Design | | seanmccauley0