Tool for Generating Sitemap/ URL List
-
HI,
I'm looking for a tool that'll generate a URL list for a site. I looked at this thread here http://www.seomoz.org/q/online-sitemap-generator which came up when I searched for sitemap generator.
However, I don't need a sitemap per se, and I don't need to submit it to Google - just a list of pages is what I need.If it updated automatically, that would be useful as well.
Anyone know of a tool, on or offline? Or anyone used Xenu and know if it's what I'm looking for? Or is there a simple solution that I'm missing?
Thanks.
-
Try DYNO Mapper. If you are looking for a tool that will generate a URL list, I think it will work really well for you. It also will give you a full content inventory that you can filter, search, and export content types. This particular Sitemap Generator is also integrated with Google Analytics and has collaboration capability for multiple users.
-
Just spent the last couple hours checking out Xenu. Found this article here on SEOmoz:
http://www.seomoz.org/blog/xenu-link-sleuth-more-than-just-a-broken-links-finder.
Turns out Xenu is an awesome tool - can't wait to make more use of it.
I also took Joyce's advice which was more suited to the task I was trying to do. Thanks for the simple solution!
-
Yes Xenu will do it, it checks all the links and then offers you a sitemap as a report at the end
I like Xenu, it's quick too
-
If you're using SEOMoz Pro, you can enter the domain into Open Site Explorer and then click the Top Pages tab. This will give you an exportable list of up to 10,000 pages on the site. Alternatively, you can go to Google Analytics and click Content > Top Content. This will give you an even bigger list, though you may have to filter out any attributes that Google picks up.
Xenu can also be a good option, but you still have to do a lot of filtering.
I hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it better to shorten my existing url to use only keyword after domain with a 301 redirect from existing url
I have a long existing URL that has included my key word but the url has about 5 additional words in the text ( eg url would have " /super widgets in stock at the widget store " as url text after domain. keywords is super widget The URL was at the top of search results for my keyword for many years until recently. Is it better to shorten my url text to now use only my keyword " /super-widgets " after the domain with a 301 direct from my existing url to optimize it Thanks
On-Page Optimization | | mrkingsley2 -
Does having a \ on a URL make a difference?
On our website we have CMS pages which end normally without a /. However when linking from web banners e.c.t it always adds a / on the end. Will this have a negative impact on SEO?
On-Page Optimization | | AHF_Furniture0 -
My home page URL http://seadwellers.com/ redirects to http://www.seadwellers.com/. Is this a problem?
"The URL http://seadwellers.com/ redirects to http://www.seadwellers.com/. Do you want to crawl http://www.seadwellers.com/ instead?" I was given this when I tried to crawl my home page using MOZ software. I was not aware of this, do not know if it could be a problem concerning any aspect of SEO, etc? :
On-Page Optimization | | sdwellers0 -
Similar URLs
I'm making a site of LSAT explanations. The content is very meaningful for LSAT students. I'm less sure the urls and headings are meaningful for Google. I'll give you an example. Here are two URLs and heading for two separate pages: http://lsathacks.com/explanations/lsat-69/logical-reasoning-1/q-10/ - LSAT 69, Logical Reasoning I, Q 10 http://lsathacks.com/explanations/lsat-69/logical-reasoning-2/q10/ - LSAT 69, Logical Reasoning II, Q10 There are two logical reasoning sections on LSAT 69. For the first url is for question 10 from section 1, the second URL is for question 10 from the second LR section. I noticed that google.com only displays 23 urls when I search "site:http://lsathacks.com". A couple of days ago it displayed over 120 (i.e. the entire site). 1. Am I hurting myself with this structure, even if it makes sense for users? 2. What could I do to avoid it? I'll eventually have thousands of pages of explanations. They'll all be very similar in terms of how I would categorize them to a human, e.g. "LSAT 52, logic games question 12" I should note that the content of each page is very different. But url, title and h1 is similar. Edit: I could, for example, add a random keyword to differentiate titles and urls (but not H1). For example: http://lsathacks.com/explanations/lsat-69/logical-reasoning-2/q10-car-efficiency/ LSAT 69, Logical Reasoning I, Q 10, Car efficiency But the url is already fairly long as is. Would that be a good idea?
On-Page Optimization | | graemeblake0 -
On Page Reports - Multiple URLs Appearing for a Keyword
Hello, I have a question regarding the on page reports automatically generated by seomoz When I look at my on page reports I notice that each keyword appears a number of times, each with a different url and then a grade for the on page report and sometimes a rank. I'm not sure I understand this, firstly I thought the on page reports were only generated for keywords in the top 50, does that mean the global top 50, or my top 50? Also why are they appearing for so many urls, I find this confusing and am not sure which pages to focus on improving, it's not always my intended pages that are ranking the best. I believe that I read somewhere that I can choose which pages to have the on page reports rank for, perhaps this is the solution? Any thoughts would be appreciated. Thanks, Iain
On-Page Optimization | | jannkuzel0 -
Can bad text URLs hurt pages?
If you have some pages that contain plain text URLs (not anchored links) that used to be good URLs, but are now bad, either because the website shut down or because it has been acquired by someone else and is now parked (or worse) - are those URLs enough to cause quality problems? For example: This information was brought to you by Waymaker http://www.waymaker.net These aren't the only ones. And yes, I know I should fix them, but there are probably 10,000 pages like it. I will fix them, but its not something I can do in a few minutes. (this one is easy to fix programmatically, but others are a lot more complex) So my question is: do you have actual experience that these are bad enough to cause ranking problems (making them low quality)
On-Page Optimization | | loopyal0 -
How many urls per page is to many
I know it used to be 100 urls per page, but recently Matt cutts has said that they can count a lot more now. I was wonder what you guys thought was how many was to many per page?
On-Page Optimization | | Gordian0 -
All Caps in URL
Hello, we're working with a corporate client to make changes to their URL structure. We recommended that they use a structure like "domain.com/state/city/location". Their IT department is on board, but they just mentioned that all of the state and city info will be pulled from a database where it is all caps. So it would be like this "domain.com/STATE/CITY/location" I'm concerned that it may be spammy, but I can't find any definitive information online. We usually like to test issues like this on our own sites before advising clients, but we're working up against a quick deadline. Any help with this issue would be great, especially real world experience, not just theories. Thank you.
On-Page Optimization | | interactivek0