Tool for Generating Sitemap/ URL List
-
HI,
I'm looking for a tool that'll generate a URL list for a site. I looked at this thread here http://www.seomoz.org/q/online-sitemap-generator which came up when I searched for sitemap generator.
However, I don't need a sitemap per se, and I don't need to submit it to Google - just a list of pages is what I need.If it updated automatically, that would be useful as well.
Anyone know of a tool, on or offline? Or anyone used Xenu and know if it's what I'm looking for? Or is there a simple solution that I'm missing?
Thanks.
-
Try DYNO Mapper. If you are looking for a tool that will generate a URL list, I think it will work really well for you. It also will give you a full content inventory that you can filter, search, and export content types. This particular Sitemap Generator is also integrated with Google Analytics and has collaboration capability for multiple users.
-
Just spent the last couple hours checking out Xenu. Found this article here on SEOmoz:
http://www.seomoz.org/blog/xenu-link-sleuth-more-than-just-a-broken-links-finder.
Turns out Xenu is an awesome tool - can't wait to make more use of it.
I also took Joyce's advice which was more suited to the task I was trying to do. Thanks for the simple solution!
-
Yes Xenu will do it, it checks all the links and then offers you a sitemap as a report at the end
I like Xenu, it's quick too
-
If you're using SEOMoz Pro, you can enter the domain into Open Site Explorer and then click the Top Pages tab. This will give you an exportable list of up to 10,000 pages on the site. Alternatively, you can go to Google Analytics and click Content > Top Content. This will give you an even bigger list, though you may have to filter out any attributes that Google picks up.
Xenu can also be a good option, but you still have to do a lot of filtering.
I hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Query for paginated URLs - Shopify
Hi there, /collections/living-room-furniture?page=2
On-Page Optimization | | williamhuynh
/collections/living-room-furniture?page=3
/collections/living-room-furniture?page=4 Is that ok to make all the above paginated URLs canonicalised with their main category /collections/living-room-furniture Also, does it needs to be noindex, follow as well? Please advice, thank you!1 -
URL Structure on Category Pages
Hi, Currently, we having the following URL Structure o our product pages: All Products Pages: www.viatrading.com/wholesale/283/All_Products.html Category Page: www.viatrading.com/wholesale/4/Clothing.html Product Page: www.viatrading.com/wholesale/product/LOAD-HE-WOM/Assorted-High-End-Women-Clothing-Lots.html?cid=4 Since we are going to use another frontend system, we are thinking about re-working on this URL Structure, using something like this: All Products Pages: www.viatrading.com/wholesale-products/ Category Page: www.viatrading.com/wholesale-products/category/ Product Page: www.viatrading.com/wholesale-products/category/product-title/ I understand this is better for SEO and user experience. However, we already have good traffic on the current URL Structure. Should we use same left-side filters on Category Pages as in All Products Page? Since we are using Faceted Navigation, when users filter the Category (e.g. Clothing) they will see same page as Clothing Category Page. Is that an issue for Duplicate Content? Since we are a wholesale company - I understand is using "/wholesale/products/" in URL for all product pages a good idea? If so, should we avoid word "wholesale" in product-title to avoid repeated word in URL? For us, SKU in URL helps the company employees and maybe some clients identify the link. However, what do you think of using the SEO-friendly product-title, and 301 redirect it to www.viatrading.com/BRTA-LN-DISHRACKS/, so 1st link is only used by company members and Canonicalized 2nd is the only one seen by general public? Thank you,
On-Page Optimization | | viatrading10 -
Need advice on the better URL structure to go with
I am rebuilding our existing website on a new platform and need advice on which URL structure would be the most ideal. The following examples are of a product that we have with a very long page title. Not all of our products have titles this long, but enough of them do to cause some concern. I was also wondering if I should end the url with file type .html or if leaving it out is better. Thanks in advance! OPTION 1. this example just uses the root domain and the entire product title separated by dashes http://ewheels.nextmp.net/staggered-full-set-br-2-20x9-ace-alloy-aff01-metallic-silver-machined-face-flow-formed-br-2-20x10-5-ace-alloy-aff01-metallic-silver-machined-face-flow-formed OPTION 2. this example uses the crawl path as well as the entire product title http://ewheels.nextmp.net/wheels/ace-alloy-wheels/ace-alloy-aff01-metallic-silver-machined-face-flow-formed/staggered-full-set-br-2-20x9-ace-alloy-aff01-metallic-silver-machined-face-flow-formed-br-2-20x10-5-ace-alloy-aff01-metallic-silver-machined-face-flow-formed OPTION 3. this example uses the crawl path and just the part number at the end since the folders already contain all the keywords necessary http://ewheels.nextmp.net/wheels/ace-alloy-wheels/ace-alloy-aff01-metallic-silver-machined-face-flow-formed/ace-2090aff01silace-20105aff01sil
On-Page Optimization | | elementmotor0 -
Lost Page Rank after directing http:// to WWW?
Hi I am trying to redirect all the non www urls to WWW. After I redirected them, most of my category page PR are dropped to 0. Can someone please tell me if this is the normal after effect after the redirect? Example url: this is PR2 before the redirect http://www.ilovebodykits.com/category/95/Body_Kits_Front_Bumpers.html
On-Page Optimization | | ilovebodykits0 -
Does 301 generate organic content ?
I manage this domain name www.jordanhundley.com . Right now it is 301 to www.jordanhundley.net where I hosted the content for almost 18 months. At this point you are only able to read the 301 script if you use CTRL U at the .com domain. Does Google read the content beyond the script? Is the 301 website getting juice from the targeted domain ? This is the script I´m using <html> <head> <title>Jordan Hundleytitle> head> <frameset rows="100%,*" border="0"> <frame src="[http://www.jordanhundley.net](view-source:http://www.jordanhundley.net/)" frameborder="0" /> frameset><noframes>noframes> html>
On-Page Optimization | | mPloria0 -
Canonical URL problem
On page analysis wanted me to add a canonical url tag. However I added then re ran the on page analysis and it came up with an error. What is the proper way to add a canonical url tag in the head of an index page? ie. add a canonical tag to www.hompeage.com/index.html would it be ? Or should I ignore this for a home page? Because I add it then run the analysis again and get this? Appropriate Use of Rel Canonical Moderate fix <dl> <dt>Canonical URL</dt> <dd>"http://www.ensoplastics.com/index.html"</dd> <dt>Explanation</dt> <dd>If the canonical tag is pointing to a different URL, engines will not count this page as the reference resource and thus, it won't have an opportunity to rank. Make sure you're targeting the right page (if this isn't it, you can reset the target above) and then change the canonical tag to reference that URL.</dd> <dt>Recommendation</dt> <dd>We check to make sure that IF you use canonical URL tags, it points to the right page. If the canonical tag points to a different URL, engines will not count this page as the reference resource and thus, it won't have an opportunity to rank. If you've not made this page the rel=canonical target, change the reference to this URL. NOTE: For pages not employing canonical URL tags, this factor does not apply.</dd> <dd>So do I add it or not? If I don't I get a lower page rating if I take it off I get a higher page rating with room for improvement. </dd> </dl>
On-Page Optimization | | ENSO0 -
Why does http://www.southernhearth.com rank so well for patio furniture and outdoor furniture?
I was looking at the competitiveness of ranking for the keywords "patio furniture" & "outdoor furniture" and was shocked to see www.southernhearth.com ranking so well. While I am just getting into the research, I was unable to see anything that would justify it's rank. What are your thoughts?
On-Page Optimization | | Mattbyo0 -
Duplicate content issue with dynamically generated url
Hi, For those who have followed my previous question, I have a similar one regarding dynamically generated urls. From this page http://www.selectcaribbean.com/listing.html the user can make a selection according to various criteria. 6 results are presented and then the user can go to the next page. I know I should probably rewrite url's such as these: http://www.selectcaribbean.com/listing.html?pageNo=1&selType=&selCity=&selPrice=&selBeds=&selTrad=&selMod=&selOcean= but since all the results presented are basically generated on the fly for the convenience of the user, I am afraid google my consider this as an attempt to generate more pages as there are pages for each individual listing. What is my solution for this? Nofollow these pages? Block them thru robots txt?
On-Page Optimization | | multilang0