Tool for Generating Sitemap/ URL List
-
HI,
I'm looking for a tool that'll generate a URL list for a site. I looked at this thread here http://www.seomoz.org/q/online-sitemap-generator which came up when I searched for sitemap generator.
However, I don't need a sitemap per se, and I don't need to submit it to Google - just a list of pages is what I need.If it updated automatically, that would be useful as well.
Anyone know of a tool, on or offline? Or anyone used Xenu and know if it's what I'm looking for? Or is there a simple solution that I'm missing?
Thanks.
-
Try DYNO Mapper. If you are looking for a tool that will generate a URL list, I think it will work really well for you. It also will give you a full content inventory that you can filter, search, and export content types. This particular Sitemap Generator is also integrated with Google Analytics and has collaboration capability for multiple users.
-
Just spent the last couple hours checking out Xenu. Found this article here on SEOmoz:
http://www.seomoz.org/blog/xenu-link-sleuth-more-than-just-a-broken-links-finder.
Turns out Xenu is an awesome tool - can't wait to make more use of it.
I also took Joyce's advice which was more suited to the task I was trying to do. Thanks for the simple solution!
-
Yes Xenu will do it, it checks all the links and then offers you a sitemap as a report at the end
I like Xenu, it's quick too
-
If you're using SEOMoz Pro, you can enter the domain into Open Site Explorer and then click the Top Pages tab. This will give you an exportable list of up to 10,000 pages on the site. Alternatively, you can go to Google Analytics and click Content > Top Content. This will give you an even bigger list, though you may have to filter out any attributes that Google picks up.
Xenu can also be a good option, but you still have to do a lot of filtering.
I hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Generating Dynamic Meta Titles for SEO - Advice?
Hi, I'm working with a new client that works with a lot of suppliers and they want to have their titles built in a specific way. If I generate a template (so to speak) will generating dynamic meta titles have a negative effect on their SEO and is this even possible? If so, what would be the best way to handle this? There is approximately 100 suppliers. Any feedback/advice is appreciated!
On-Page Optimization | | daniel-brooks0 -
With 301 Redirects Does Changing URLs Matter?
We are redesigning our website in order to give it a more modern visual look. For the most part all the content will remain the same. Our old site is hosted on .asp so all of our current URLs look something like this: www.example.com/products/food.asp We plan on using 301 redirects in order to update every URL and remove the .asp. Since we are going to be doing 301 redirects for every existing URL anyways, does it matter from an SEO and ranking standpoint, if we also change the content and structure of the URL? For example, would we see a ranking impact if we were to change the above example URL to www.example.com/food? Obviously we want to try to retain as much link juice and ranking factors as possible during this redesign. Another issue we are seeing is with the image file names of our existing website images. We are moving to a new CMS platform (WordPress) that automatically saves images using a folder path similar to this: wp-uploads/2015-08/food. Will that change affect our SEO or ranking at all? When Google crawls an image does it care about the full path? Any insight would be much appreciated! 🙂
On-Page Optimization | | BlueLinkERP0 -
Submitting URLs to Bing and Google
Does Submitting URLs to Bing and Google actually do anything? Is it worthwhile? What I mean is submitting intermittently individual URLS after already submitting the sitemap.
On-Page Optimization | | FCAbroad0 -
Rewrite URL for big website
My website is currently have over 1000 000 links indexed by Google , i'm going to rewrite all of these links, is this crazy ? Can someone give me a checklist to avoid going wrong way ? My website: http://www.webtretho.com/forum/ My current url format: webtretho.com/forum/f[forum_id]/[thread-title-thread_id/ Ex: Old URL: http://www.webtretho.com/forum/f90/chia-se-dau-hieu-mang-thai-1357342-new/ New URL: http://www.webtretho.com/forum/chuan-bi-mang-thai/chia-se-dau-hieu-mang-thai-1357342-new/
On-Page Optimization | | firstjames0 -
Canonical URL problem
On page analysis wanted me to add a canonical url tag. However I added then re ran the on page analysis and it came up with an error. What is the proper way to add a canonical url tag in the head of an index page? ie. add a canonical tag to www.hompeage.com/index.html would it be ? Or should I ignore this for a home page? Because I add it then run the analysis again and get this? Appropriate Use of Rel Canonical Moderate fix <dl> <dt>Canonical URL</dt> <dd>"http://www.ensoplastics.com/index.html"</dd> <dt>Explanation</dt> <dd>If the canonical tag is pointing to a different URL, engines will not count this page as the reference resource and thus, it won't have an opportunity to rank. Make sure you're targeting the right page (if this isn't it, you can reset the target above) and then change the canonical tag to reference that URL.</dd> <dt>Recommendation</dt> <dd>We check to make sure that IF you use canonical URL tags, it points to the right page. If the canonical tag points to a different URL, engines will not count this page as the reference resource and thus, it won't have an opportunity to rank. If you've not made this page the rel=canonical target, change the reference to this URL. NOTE: For pages not employing canonical URL tags, this factor does not apply.</dd> <dd>So do I add it or not? If I don't I get a lower page rating if I take it off I get a higher page rating with room for improvement. </dd> </dl>
On-Page Optimization | | ENSO0 -
XML sitemap latest best practices?
If anyone optimised their sitemap recently and would like to share some tips it would be much appreciated:-) Cheers
On-Page Optimization | | dublinbet0 -
URL Strucutre
Hi there, Need some advice please on URL structure. I have been doing SEO for quite sometime now, however one thing that always get me is URL structure. I have a decision to make, its either: URL 1 /conditions/allergies/food/ URL 2 /conditions/allergies-food/ Lets say i am optimizing for the key-phase "Food Allergies" what do you think is best practice? I know that this is not a major factor in gaining high SERPs & maybe i'm thinking about it too much, however your input would be really helpful. Kind Regards,
On-Page Optimization | | Paul780 -
Duplicate content issue with dynamically generated url
Hi, For those who have followed my previous question, I have a similar one regarding dynamically generated urls. From this page http://www.selectcaribbean.com/listing.html the user can make a selection according to various criteria. 6 results are presented and then the user can go to the next page. I know I should probably rewrite url's such as these: http://www.selectcaribbean.com/listing.html?pageNo=1&selType=&selCity=&selPrice=&selBeds=&selTrad=&selMod=&selOcean= but since all the results presented are basically generated on the fly for the convenience of the user, I am afraid google my consider this as an attempt to generate more pages as there are pages for each individual listing. What is my solution for this? Nofollow these pages? Block them thru robots txt?
On-Page Optimization | | multilang0