Can SEO Moz perform a full site crawl and provide a report showing all URLs within an existing domain?
-
We are conducting a site redesign and need to get an idea of all pages that are out there on our domain (in some report fashion). This would help for discovery and cleanup as we re-work the site and move to a new CMS.
Thanks
-
Hey Stephen,
THanks - I have heard of Xenu many times before. I'll try that out.
I appreciate the input!
-
just going to throw in an alternative for this type of thing Xenu Link sleuth - free, easy and powerful.
Yes that site looks like ass but the tool is great. Dr Pete and others have done "how to use" posts on seomoz - just search xenu on seomoz
-
Yes - that's exactly it. Can I just do an export of the campaign crawl? or is that found somewhere else? Forgive my ignorance...still new to the tool and just can't seem to find.
-
Are you looking at your crawl report? It is available in an Excel file which provides the exact information you desire.
-
Hey Ryan,
Thanks for the response.
It's not a huge site. I would say there's probably around 600 total pages. It definitely looks like the SEO crawler has crawled the entire site, but I can't seem to find a way to just get a one-shot view of all the pages it's accessed.
Any ideas on where this can be found?
Thanks again!
-
How many pages are on your site?
The SEOmoz crawler will do exactly what you want but there is a URL limit. You should also understand it will experience the same limitations of any crawler. It wont list pages which behind a user log-in nor pages which are blocked by robots.txt. I am pretty sure the SEOmoz crawler doesn't list secure pages either but last I heard they were working on including them.
If your site has any island pages they wont be discovered either.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEO Ranking: Can Child Theme Compete with Custom Theme?
Ranking for New York City commercial real estate is extremely competitive. We compete against: www.squarefoot.com, www.42floors.com, www.Loopnet.com, www.wework.com and a dozen other optimized sites. Our site was designed in 2012. We plan on upgrading it. From an SEO perspective, can we compete by purchasing a Wordpress real estate theme and customizing it into a child theme? Our better ranking competitors are using custom themes where the code has been very streamlined to make the sites quick and easy to index by spiders. Would we gain a significant edge by custom coding? This is somewhat technical for a business owner and I am trying to get my head around it. Our existing site is www.nyc-officespace-leader.com. Some of the themes we are considering are: -http://main.wpestatetheme.org/homepage -http://houzez01.favethemes.com/ -http://realhomes.inspirythemes.biz/property/ From an SEO perspective is creating a child theme from the above a good approach? Or will a custom theme give us an advantage. If there is an advantage is that edge so marginal that it is not significant? In terms of coding, is a custom site much more labor, 2x, 3x the time to code? Also is the maintenance of a custom site much more involved? Also, as a related question, my developer since 2012 has created many custom plugins for Wordpress. Is this a no, no? Will avoiding custom plugins add to the development cost? Even if we use a child theme from an existing real estate website, I would hope that the improved user interface will provide a boast in at least conversions if not SEO. Thanks, Alan
Web Design | | Kingalan10 -
How does adding ecommerce to a site affect SEO? What are the negative and what are the positives?
We are thinking of adding ecommerce to our website as a service to our customers. We generate most of our leads through online quote requests but heard that it may be beneficial to our SEO if we add ecommerce for a few products. Is this true? Does anyone have tips on best and worst SEO ecommerce practices?
Web Design | | TeguarMarketing0 -
Average Time to Conversion on Site
I am curious to know if there is a way to view or calculate the average time it takes site visitors to convert per session. For example, based on a current website design, the average time on site might be 3 minutes and the number of conversions might be 100. is there a way to say that for the current website design, it takes 3 minutes for the average site visitor to submit a web form? Then, as I redesign the site, my goal would be to improve the average time to conversion by making the web form more accessible and require less information within the form itself. I don't think this is currently possible in GA. Has anyone figured out a way to accomplish this by use of traditional tracking tools? Or, am I facing having to code my site to record each visitor's time on site from the second they enter and then stop the clock when they submit the form?
Web Design | | dsinger0 -
Totally flat URL structure
Hi Mozzers! I've just been viewing a website with a flat URL structure - the site has a definite structure - with various sections - and yet the URL structure doesn't reflect this... The developer tells me this is purely for SEO purposes! Would be interested in your thoughts...
Web Design | | McTaggart0 -
URL Designing and Site Architecture
Hi, 1. At the end of the URL, should we have "/" or not (for e.g. my website www.example.com/abc/) 2. What is the optimum level of site depth 3. Google crawler will go through the breadcrumb or url (If a page is at 6th level through breadcrumb and at 4th level through URL, at which level will google crawler recognize this.
Web Design | | adiez12340 -
Two URLs with same content
We recently had a client who own multiple brands switch from having multiple urls to having a single domain with multiple sub domains. I've posted an example below to better explain. My question is the original url is still functional, so there are two urls with identical content, yet I haven't been getting a duplicate content error. Also, would a rel canonical link be beneficial in this case since the duplicate content is on two separate domains? My thoughts were to put a 301 redirect on the original pages so they permanently forward to the new sub-domain format. Is this the best course of action? If not, what would you recommend? Example: Original URLs
Web Design | | BluespaceCreative
www.example1.com
www.example2.com
www.example3.com
www.parentcompany.com New URLs
example1.parentcompany.com
example2.parentcompany.com
example3.parentcompany.com
www.parentcompany.com Let me know if this I need to clarify anything in better detail.
Thanks in advance!0 -
Can "poor" subdomains drop PR of the root domain?
The page rank of my company's website has dropped from a 6 to a 4 over the past year or so. In that time, we implemented subdomains for development sites to show clients progress on their websites. I noticed that our "dev" sites are being indexed while in development and my question is, will Google drop pagerank of our root domain purely off of these "dev" subdomains? Example - our site is www.oursite.com Dev site - development1.oursite.com I just began investigating the drop and this came to my mind yesterday but am not too sure what type of impact these non-credible subdomains will have on our root domain. Any thoughts?
Web Design | | ckilgore0 -
Crawl Budget vs Canonical
Got a debate raging here and I figured I'd ask for opinions. We have our websites structured as site/category/product This is fine for URL keywords, etc. We also use this for breadcrumbs. The problem is that we have multiple categories into which a category fits. So "product" could also be at site/cat1/product
Web Design | | Highland
site/cat2/product
site/cat3/product Obviously this produces duplicate content. There's no reason why it couldn't live under 1 URL but it would take some time and effort to do so (time we don't necessarily have). As such, we're applying the canonical band-aid and calling it good. My problem is that I think this will still kill our crawl budget (this is not an insignificant number of pages we're talking about). In some cases the duplicate pages are bloating a site by 500%. So what say you all? Do we just simply do canonical and call it good or do we need to take into account the crawl budget and actually remove the duplicate pages. Or am I totally off base and canonical solves the crawl budget issue as well?0