Can SEO Moz perform a full site crawl and provide a report showing all URLs within an existing domain?
-
We are conducting a site redesign and need to get an idea of all pages that are out there on our domain (in some report fashion). This would help for discovery and cleanup as we re-work the site and move to a new CMS.
Thanks
-
Hey Stephen,
THanks - I have heard of Xenu many times before. I'll try that out.
I appreciate the input!
-
just going to throw in an alternative for this type of thing Xenu Link sleuth - free, easy and powerful.
Yes that site looks like ass but the tool is great. Dr Pete and others have done "how to use" posts on seomoz - just search xenu on seomoz
-
Yes - that's exactly it. Can I just do an export of the campaign crawl? or is that found somewhere else? Forgive my ignorance...still new to the tool and just can't seem to find.
-
Are you looking at your crawl report? It is available in an Excel file which provides the exact information you desire.
-
Hey Ryan,
Thanks for the response.
It's not a huge site. I would say there's probably around 600 total pages. It definitely looks like the SEO crawler has crawled the entire site, but I can't seem to find a way to just get a one-shot view of all the pages it's accessed.
Any ideas on where this can be found?
Thanks again!
-
How many pages are on your site?
The SEOmoz crawler will do exactly what you want but there is a URL limit. You should also understand it will experience the same limitations of any crawler. It wont list pages which behind a user log-in nor pages which are blocked by robots.txt. I am pretty sure the SEOmoz crawler doesn't list secure pages either but last I heard they were working on including them.
If your site has any island pages they wont be discovered either.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can the design still be considered adaptive if the URL is different?
I was under the impression our site had a mobile dedicated design, but my developers are telling me we have an adaptive design. The mobile site is set up different and has different content and the url is as follows: www.site.com/MobileView/MobileHome.aspx Can it still be considered adaptive if the URL is not the exact same? Hopefully this make sense and I appreciate anyone's input!
Web Design | | AliMac260 -
How to export Wordpress comments ONLY to a new domain
Hi Guys, We have a bit of situation here. We have a website (let's say it is www.oldsite.com) where we had more than 2000 posts. There arose a need whereby we had to move some 60-70 posts from this oldsite.com to another domain of ours (www.newsite.com). So, here is what we did: Move those 60-70 posts manually from oldsite.com to newsite.com Did a 301 redirect of each of those 60-70 posts from oldsite.com to newsite.com. Google has now started to rank the posts from the newsite.com for this. That's all good till now. Now, here comes the situation. We also want to move the comments from some of those posts of oldsite.com (some 10-12 posts out of those 60-70) to the respective posts of newsite.com. How do we do that? Do note that we are pretty comfortable with databases and to some extent PHP. Please help.
Web Design | | seocuppa0 -
SSL, SEO, and Site Migration question
When migrating a site to a new url and one where the old url had no https and the new url will be full https does it matter if the 301 redirect points at http://thisisthenewsite.com ? Meaning, should the new site have the ssl / https up prior to redirecting the old site? Does it matter if you redirect the old site to http://thisisthenewsite.com or https://thisisthenewsite.com? Since the site will force to https anyway?
Web Design | | Atlanta-SMO0 -
Two sites in same industry and which shopping cart
Right. So I suspect I am going to sound paranoid here - but you'll all forgive me right?? I am sure I saw a reply to a question on the Q&A suggesting that it was a bad idea to have two sites in one industry as Google may see it as trying to get two bites of the SERP cherry... is this accurate? I have an existing asp.net site in the maternity wear industry here in Australia and am wanting to start another site to appeal to a different customer base... the market is quite broad. There will be a core list of products that are the same between the sites, but also some quite different products. Content, product descriptions and categorys will be different. I have another website that I bought with reasonable age and links in the industry that I was going to 301 to the new site to give it a kick in the juice. So, not wanting to deceive my customers in anyway, I was thinking I would call it a "division of" or "sister site to" the existing ecommerce site, with a single link back and forward between the two sites. Would there be anything wrong with this in googles eyes? Even with same contact details? They would be run on totally different platforms and hosted by totally different providers. Or would you keep them totally seperate and only have contact details in images? Or a step further and have totally different phone numbers etc? Then the shopping cart - I would love some suggestions on which opensourse cart to use, preferrably one that I can set up myself, and that has a good framework for seo. I want to use schema.org, authorship, seo friendly urls all of which I am having trouble getting out of the developer of my asp.net site.... I don't want the new site to be asp.net Thanks in advance!!
Web Design | | catfree0 -
Turning my Design Business site into a site to promote SEO
I need advice on retooling my website for my SEO biz. I have shifted my business model from graphic designer who does websites, to "internet marketing consultant who does graphics too". My main website and domain name is over 10 years old, so I've made the decision to keep it, even though it has no keywords in the name. The name works well for the new business, otherwise. The site has a PR3 and I rank well for small business advertising terms, which gets me graphic design business. I intend to keep doing graphic design, but that is a smaller part of my income. I had considered making 3 satellite sites with keyword domain names to cover my offerings of graphic design SEO, website development, and internet marketing. But am leaning against it for several reasons (that all of us SEO's know) but mainly the fact that I cannot keep up with both working for my clients and blogging on multiple sites and link building for multiple sites. So my question is (you knew there was one coming, right?), what is the best approach to building categories of web development, internet marketing, and SEO into my existing graphic design/advertising oriented website? This is slightly embarrassing to ask as an SEO, but given the multiple approaches possible, and knowing the importance of doing it right the first time, it's best to get an consensus perspective on the BEST approach. My main concerns are the navigation system and the links from the homepage into the site. I have too many pages I've identified as essential to link off of the home page and navigation menus? (Website development, social media marketing, link building, keyword research, pay per click, online advertising, graphic design, brochures, catalogs, Logos, Branding, SEO, keyword research etc.) I've always tried for the ratio of one link off of any page for every 100 words of content. Do I create a home page that is of monster proportions? Do I just have the 4 basic areas linking off the home page then create a "landing zone" of 4 folders and create down from that? I am concerned about URL length as I go deeper with that approach. Or, does it make more sense to have a dozen second-level pages, and not link them all off the home page, and build from beneath (and relying on external juice). Next issue is the nav system. It will be huge. Am I best off just keeping it to 4-6, and creating subnavigation on everypage within the site according to section (PITA)? I've read dozens of blog opinions on how much nav systems do or do not hurt link juice. I've always thought footer links were right next to worthless to pass any juice, but given this situation, does it make sense to make a footer link for each major page (about 20)? Thanks for your opinions.
Web Design | | JCDenver0 -
What site do you admire/like for its SEO - technical, content, whatever - and why?
I am gathering examples of great SEO'd sites and would appreciate your examples. The rationale can be anything - great SEO structure, great linking, solid content - you think stands out. Thank you!
Web Design | | josh-riley0 -
Development site accidentally crawled - Will this cause problems?
We are currently developing a new version of our website and to make it easy to access for all team members, we just set it up on a server accessible via a publicly accessible domain name (ie devsite.com). There has been no SEO and no links created to this site, or so I thought. Recently, I found out that Google somehow found its way to this development site and has been indexing the pages! I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain. I immediately created a robots.txt file to disallow access to the entire development domain. My fear is that there may be some duplicate content penalty if Google sees that the content that is on our new site (once it goes live and is pushed to our REAL domain name) was previously indexed on our test domain. We're slated to launch in 2-3 weeks. Is there anything else I should do? Should I even be worried? I'm probably a bit paranoid, but given the amount of time and effort that has gone into this new site, I love any advice or thoughts. Thank You!
Web Design | | AndrewY0 -
Finally have a budget for a great seo ecommerce site but need help choosing wordpress, joomla, modx, magneto or? Thank you in advance for your generosity of time
We finally have a budget and want to dump our intuit/homestead site www.originalartbroker.com Our budget is 5k-10k but could do more if needed. I am slowly catching my competition with this homestead site that I built. But I do realize it is time to step back, figure out what is best, and hire a pro to get the job done. I am green in the seo and web development arena so please go easy on me and please help to point me in the right direction. Just went out on a limb a couple years ago playing with homestead.com site software and built what we have today. Didn't know anything about website development...it sort of just happened. I feel and know that homestead.com solution is hindering what we could be doing due to the bloated nature of the site and inability to perform such task as 301 redirects etc.... I have been able to slowly attain first page seo rankings on keywords based of the artists we carry using this po-dunk homestead platform to build my site after a ton of work education thanks to seomoz and a lot of you. But, have never asked for help and could really use some generosity of time in explaining a solution that would work best for our business. Do we just go with a wordpress site that is similar to our current setup and use their plugins? Do we use a cms software solution like magneto or joomla? We will only have 200-300 pieces at any given time. We are constantly selling and buying new pieces providing us content. We are need of a site that can perform well in terms of seo. I have heard of a lot of people talking about joomla, wordpress, and magneto. Would like to be able to have a product catalogue that ultimately sends whatever inventory we are uploading to our social sites and blogs so I don’t have to pump the product out to all of these sites. We offer free custom framing with our pieces and it would be nice to have a program that could wrap the photos of the pieces with the different frames for our customers. When I add a new piece I would like this software to have a predesigned product page that it plugs the information into. I would like it to create the url extension based of the artists name, medium used, and piece name to create unique and individual urls. I would like it to also create its own H tags throughout that product page according to the artist name description, and medium used. I would like to be able to sink this up to google merchant and other sites to carry our product. Bottom line is we sell art. We sell pieces by specific artists. We are constantly buying and selling. I need something powerful that keeps up with our content
Web Design | | forecastedinvestments1