Can SEO Moz perform a full site crawl and provide a report showing all URLs within an existing domain?
-
We are conducting a site redesign and need to get an idea of all pages that are out there on our domain (in some report fashion). This would help for discovery and cleanup as we re-work the site and move to a new CMS.
Thanks
-
Hey Stephen,
THanks - I have heard of Xenu many times before. I'll try that out.
I appreciate the input!
-
just going to throw in an alternative for this type of thing Xenu Link sleuth - free, easy and powerful.
Yes that site looks like ass but the tool is great. Dr Pete and others have done "how to use" posts on seomoz - just search xenu on seomoz
-
Yes - that's exactly it. Can I just do an export of the campaign crawl? or is that found somewhere else? Forgive my ignorance...still new to the tool and just can't seem to find.
-
Are you looking at your crawl report? It is available in an Excel file which provides the exact information you desire.
-
Hey Ryan,
Thanks for the response.
It's not a huge site. I would say there's probably around 600 total pages. It definitely looks like the SEO crawler has crawled the entire site, but I can't seem to find a way to just get a one-shot view of all the pages it's accessed.
Any ideas on where this can be found?
Thanks again!
-
How many pages are on your site?
The SEOmoz crawler will do exactly what you want but there is a URL limit. You should also understand it will experience the same limitations of any crawler. It wont list pages which behind a user log-in nor pages which are blocked by robots.txt. I am pretty sure the SEOmoz crawler doesn't list secure pages either but last I heard they were working on including them.
If your site has any island pages they wont be discovered either.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are provincial third-level domains bad for SEO?
My prospect's domain ends in ".on.ca" (Ontario, Canada). The structure of their site is companyname.on.ca (main page) and all other pages are sub-folders (companyname.on.ca/page-name-1 All pages are no more than two levels deep. I'm wondering if anyone knows if the provincial sub-domain (.on.ca) presents an SEO challenge?
Web Design | | 22Eighteen1 -
Ecommerce site in Europe
A client is looking to expand their ecommerce business into Europe. They have already purchased a number of European brand domain names and want to start their expansion in Ireland. Is it better to have a single website with pages translated for each language or a seperate site for each domain/country? With Ireland the only obvious difference is the currency, language shouldn't be an issue. But if we choose seperate sites for each domain/country will having the same product content/descriptions especially in the case of Ireland be an issue? Thanks in advance
Web Design | | MartinJC0 -
URL Designing and Site Architecture
Hi, 1. At the end of the URL, should we have "/" or not (for e.g. my website www.example.com/abc/) 2. What is the optimum level of site depth 3. Google crawler will go through the breadcrumb or url (If a page is at 6th level through breadcrumb and at 4th level through URL, at which level will google crawler recognize this.
Web Design | | adiez12340 -
Confluence and SEO
I think this is a difficult question so apologies in advance and any help would be appreciated! We currently have a large amount of support center content sitting on our main pages which we don’t think is very effective (mainly basic how to guides). We think it is difficult for visitors to understand and the UI is very poor. In order to solve this we’re currently moving this content onto a subdomain using Confluence, a wiki based team collaboration tool (from a company called Atlassian). What we’re planning on doing is very much like what Atlassian themselves have done on this page: https://confluence.atlassian.com/display/ALLDOC/Atlassian+Documentation What are the SEO issues / dangers that I need to consider before moving this content? I’m assuming that as this content will still be on the same domain then we can minimise link equity / authority loss by setting up re-directs to the new content. Also, has anyone had any experience of using Confluence and whether individual pages can be optimised for SEO? I notice that there are lots of add-ins that can be used, one of which is an SEO add-on which allows you to customise things like meta description tags.
Web Design | | RG_SEO0 -
The Best Wordpress Templates and The Best SEO Plugins??
Hey guys, I am a newbie and after so many painstaking hours of building my new site using the WIX platform, I just found out that WIX is not SEO friendly 😞 So referring to (copy n pasted) the old post below posted in by Big Bazza back in June 2011, does anyone agree or have any updated information they can assist me with so I can get my new site started ? Cheers 🙂 www.studiopress.com - excellent templates and support is first class! EDIT: These all support WP 3.0 and are SEO friendly too! (They have modified the SEO functionality) However, the framework also allows you to run Yoast Wordpress SEO - which is generally considered the best SEO plugin for WP. http://yoast.com/wordpress/seo/
Web Design | | Giorgio680 -
What is wrong with our site?
Hello Seomoz friends. I've about to pull all my hair out and need to turn somewhere. Our site, www.sightmax.com has been around since 2004. We used to be ranked at the top of page two on google under the keyword "live chat". We are no on page 4, heading to page 5. Can anyone take a look at the site and see if anything jumps out at you? The only way we have been able to get on the first page, is the pay for google adwords (which we've been doing every month for 7 years). Again, the site www.sightmax.com and the keyword is "live chat". Any help or feedback would be appreciated SO MUCH! Thanks! Eric
Web Design | | EricWeber0 -
Parked Domain or Redirect
Should I park a domain or Redirect? And what is the best way? I need to switch our domain name. I currently have all of our domains redirecting to our main website. I have set up there own hosting in our cpanel account so I could redirect them to our main domain. Was this too many steps? I tried putting all of our domains in our main domains, .htaccess file, and redirected them to our main website, but they did not work. So that is why I set up there own cpanel accounts. Now they work fine. However, my hosting company told me that I could just park the domain on our current domains account. If I can redirect all of these domain in one place, that would be great. I thought that a parked domain is considered duplicate websites, as both urls work, displaying the entire website with both urls. So Would I have to re upload our entire website to the account that I want as our main domain? Or is there another way of going about doing this?
Web Design | | hfranz0 -
SEO to Drive Leads
Dear friends, I have clients asking for SEO to drive leads. I have no experience on it, how could I do that? Is there any web layout I can suggest to my client? Another question is how do I implemented the layout on the web structure? I usually structuring my website based on keywords. For example: Homepage niche: plumbing Category: london plumbing, plumbing service, etc Content: Finding London plumbing service in your area, what kind of plumbing service you should pay attention? So, based on these layout, my SEO efforts goes to the content, instead of the homepage. Now, the question is how do I increase conversion with these strategy? Is placing a form above the fold would be wise, then place the content below it. Please advice 🙂 Thanks for your kind support, Albertus
Web Design | | toeplly0