Can SEO Moz perform a full site crawl and provide a report showing all URLs within an existing domain?
-
We are conducting a site redesign and need to get an idea of all pages that are out there on our domain (in some report fashion). This would help for discovery and cleanup as we re-work the site and move to a new CMS.
Thanks
-
Hey Stephen,
THanks - I have heard of Xenu many times before. I'll try that out.
I appreciate the input!
-
just going to throw in an alternative for this type of thing Xenu Link sleuth - free, easy and powerful.
Yes that site looks like ass but the tool is great. Dr Pete and others have done "how to use" posts on seomoz - just search xenu on seomoz
-
Yes - that's exactly it. Can I just do an export of the campaign crawl? or is that found somewhere else? Forgive my ignorance...still new to the tool and just can't seem to find.
-
Are you looking at your crawl report? It is available in an Excel file which provides the exact information you desire.
-
Hey Ryan,
Thanks for the response.
It's not a huge site. I would say there's probably around 600 total pages. It definitely looks like the SEO crawler has crawled the entire site, but I can't seem to find a way to just get a one-shot view of all the pages it's accessed.
Any ideas on where this can be found?
Thanks again!
-
How many pages are on your site?
The SEOmoz crawler will do exactly what you want but there is a URL limit. You should also understand it will experience the same limitations of any crawler. It wont list pages which behind a user log-in nor pages which are blocked by robots.txt. I am pretty sure the SEOmoz crawler doesn't list secure pages either but last I heard they were working on including them.
If your site has any island pages they wont be discovered either.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moving pages to new domain
Hello, Our product pages are ranked #1 on google for our target keywords using our domain e.g. www.olddomain.com/cases/productxyz and sell about 20 products all ranked #1. We have a new company called www.newco.com/case/product1, 2, 3 etc. We use woocommerce e-commerce for both old and new sites. What is the best way to list our old co-products on our new site and move over the #1 rankings? Do we create new products (using our new nice design) in the newco.com woo commerce and then redirect old co links? do we copy and paste all that old content into the newco.com? Totally confused. Thank you!
Web Design | | Jamesmcd031 -
Client Portal and SEO Considerations?
Hi Moz and Moz fans, We are looking to add a client portal to the website. Basically, I haven't found too much on this with regard to SEO. The idea would be that certain parts of the website would be hidden under a pay wall and for subscribers, they would be able to see all content. I am wondering if anyone has any experience with that and what SEO considerations to take into account. One thing we are particularly concerned about is how Google will index the portions of the website behind the pay wall, if at all. Obviously, we would rather that they don't index it, so that people can't find a way to get to the info without paying. I would imagine it would have to do with the type of coding, however, I am not a coding guru, so I am not 100% on that. Anyway, anyone that has any experience in this kind of thing and can comment on this at all, any comment is welcome. Also, any documentation that could be helpful would be welcome too. Thanks
Web Design | | Brian_Dowd0 -
URL Designing and Site Architecture
Hi, 1. At the end of the URL, should we have "/" or not (for e.g. my website www.example.com/abc/) 2. What is the optimum level of site depth 3. Google crawler will go through the breadcrumb or url (If a page is at 6th level through breadcrumb and at 4th level through URL, at which level will google crawler recognize this.
Web Design | | adiez12340 -
Site is losing traffic after relaunch
Hello, We've just relaunched this site in the last several days, and we're seeing some small (but stead) traffic decreases, as well as engagement decreases. We're aware that page speed (about 4 seconds from a non-cached browser) and some 404s are an issue, our team is currently working on both. But we're really looking for some constructive criticism here as to what we need to improve. Other issues to be aware of: lots of our social counts went back to 0s, as lots of URLs changed, and it wasn't possible to migrate comments from the old system, so those have gone back to 0 as well. We wonder if this might be affecting both users and search engines perception of the site. Your input would be greatly appreciated. Thanks,
Web Design | | FishAcct
Paul0 -
Keywords in the page url for best SEO
Hello all, I am working in the keywors structure of a web and I have the following doubt: If I want to target these keywords: great food madrid and my website is: http://www.madridlive.com I do not know if I should keep either: OPTION 1: page url: www.madridlive.com/great-food-madrid or OPTION 2: page url www.madridlive.com/great-food I do not know if the search engines "understands" madrid in "madridlive", therefore I can avoid the "madrid" keyword, dicarding option 1 and going for option 2. Additionally I avoid duplication of the madrid keyword that can be seen as redundancy and also have a shorter page url. Thank you very much and sorry for such a question but I am new in this SEO field...just the excellent SEOMOZ's SEO Guide for beginners! Best regards, Antonio
Web Design | | aalcocer20030 -
Effect of Off-Site Images
I'm getting to start work with a new client, and I've run across something I've never had to deal with before, off-site images. The site I'll be working on is for an appliance retailer, both online and physical. The way they've had their site built (not something I was part of) a third party company maintains the product inventory side of things. They're sourcing from about 35 different manufacturers, and this third party has direct access to the product information streams. They push the weekly updated information to my clients site. What this means, though, is that the product images don't live on the client's site. They're hotlinked from the third party's inventory doohickey. I've never seen something quite like this before. Has anyone else? Any ideas as to what problems I may face when it comes to on-site SEO?
Web Design | | MRCSearch0 -
Are tables OK in e-commerce sites?
I know that for best results with search engines, tables should not be used. However as I look to redesign our older e-commerce site, I find that not only our website but many other e-commerce sites use tables. Is it acceptable to use tables in e-commerce or am I just viewing outdated e-commerce sites. Could you recommend a good shopping cart solution? We currently use volusion. Thanks for your time and see you in Boston next week!
Web Design | | Lael0 -
Changing the domain - To do or not to do - that is the question
Hello, I am in the process of updating my website (hopefully to improve SEO). It is a major update as we are going from 20,000 product line to 200,000 product lines and hiring two marketing people to work on more content. Unfortunately, I think my domain isn't the best i.e. www.direct-tractorparts.com Some of the parts have long part numbers and I am worried that my URLs will be too long i.e. I will lose link juice from the search engines. typical url www.direct-tractorparts.com/6XY41210TTTAZCAz2 I am thinking of launching the new site with a shorter domain name rather than updating the old site. My site doesn't have that many backlinks but generates a decent level of business for my company. Not sure if it would be worth setting up another brand or not Anyone got any advice?
Web Design | | DavidLenehan0