Best server-side sitemap generators
-
I've been looking into sitemap generators recently and have got a good knowledge of what creating a sitemap for a small website of below 500 URLs involves. I have successfully generated a sitemap for a very small site, but I’m trying to work out the best way of crawling a large site with millions of URLs.
I’ve decided that the best way to crawl such a large number of URLs is to use a server side sitemap, but this is an area that doesn’t seem to be covered in detail on SEO blogs / forums. Could anyone recommend a good server side sitemap generator? What do you think of the automated offerings from Google and Bing? I’ve found a list of server side sitemap generators from Google, but I can’t see any way to choose between them. I realise that a lot will depend on the type of technologies we use server side, but I'm afraid that I don't know them at this time.
-
Unless they have fixed it in recent months, xml-sitemaps does not generate correct video sitemaps.
-
Yeah, they offer free and paid hosted versions too. But I found the server side version much simpler to setup and control.
-
-
Excellent advice Federico. My first reaction was, "but that's not a server-side sitemap generator". I just looked at their website though and it turns out that it is! Looks like I need to read things more carefully!
I'll look into that as an option but if anyone else has any server side sitemap generators that they'd recommend then I'd be really interested to hear about them
-
I have been using xml-sitemaps (paid version) for all my sites over 5 years and they work like a charm, scraping and indexing what it needs to be indexed ans scraped, plus it consumes really low resources. 100% recommended (they have nice plugins too for extra sitempas (video, news, images, etc).
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Whats the best practice for acquisition?
Hi, My company have just bought out a competitor. We wan't to dissolve their website and if possible steal some of their link juice. The site hasn't got any spammy links or 404's so i'm not worried in that department. What I am not sure about is which of the following is best practice? a. Redirect every single page (even pages like /?checkout) to a relevant page on our website. b. Only redirect important pages, category pages, contact pages etc and leave the other pages to 404? c. Redirect the important pages to a relevant URL and redirect the less important pages to our homepage. d. Redirect the entire domain to our home page (i assume this isn't a good idea) e. Don't redirect any of the pages just delete the site.
Intermediate & Advanced SEO | | DannyHoodless0 -
Best tips for getting a video page to rank?
We have a video for our company, located here: http://www.imageworkscreative.com/imageworks-creative-video It's an overview of our company and the services we offer. We'd like to get this page ranking, but we haven't had much luck so far. Our Youtube account does better, but I'm looking for some things we can do on or offsite to get this page to rank. Any tips would be appreciated!
Intermediate & Advanced SEO | | ScottImageWorks0 -
Best anchor text strategy for embeddable content
Hi all We provide online services, and as part of this we provide our clients with a javascript embeddable 'widget' to place on their website. This is fairyly popular (100s-1000s of inserts on websites). The main workings of this are javascript (they spit html iframe onto the page) but we also include both a <noscript>portion (which is purely customer focused, it deep links into a relevant page on our website for the user to follow) and also a plain <p><a href=''></a></p> at the bottom, under the JS. This is all generated and inserted by the website owner. Therefore, after insertion we can dynamically update whatever the Javascript renders out, but the <noscript> and <a> at the bottom are there forever.</p> <p>Previously, this last plain link has been used for optimisation, with it randomly selecting 1 out of a bank of 3 different link anchor texts when the widget html is first generated.</p> <p>We've also recently split our website into B2B and B2C portions, so this will be linking to a newer domain with much established backlinks than the existing domain. I think we could get away with optimised keyword links on the old domain but the newer domain they will be more obvious.</p> <p>In light of recent G updates, we're afraid this may look spammy. We obviously want to utilise the link as best as possible, as it is used by hundreds of our clients, but don't want it to cause any issues. </p> <p>So my question, would you just focus on using brand name anchor text for this? Or could we mix it up with a few keyword optimised links also? If so, what sort of ratio would you suggest?</p> <p>Many thanks</p></noscript>
Intermediate & Advanced SEO | | benseb0 -
Best Keywords for my local niche
Hello, I'm a health coach helping people with multiple sclerosis. Here's my website: bobweikel(dot)com What do you think the top 4 local keywords would be for my niche? I'm in Boise ID. I'm thinking MS Boise MS Boise Idaho Multiple Sclerosis Boise Multiple Sclerosis Boise Idaho With your intuition, do you think these are valuable keywords for a coaching site? Also, can you think of any other keywords? I want this 100% white hat (no cloaking, for instance)
Intermediate & Advanced SEO | | BobGW0 -
List of Best SEO forums
Could I please get some input on a list of the best SEO forums out there besides SEOmoz? (Both mainstream and non mainstream)
Intermediate & Advanced SEO | | Luia0 -
Multiple sitemaps for one site?
Excuse my sitemap ignorance here. I've got a site and it's got a blog in a sub-folder. The blog gets updated frequently, the main site does not. Is it best to; a) Have 2 sitemaps.. one in the root and one in the /blog folder. b) Have 1 sitemap that is regularly updated The reason being, I know there's various plugins that create blog sitemaps on the fly, so that would be much easier than updating the main sitemap every time a change was made. If the answer is 2 sitemaps; Would you stop the root sitemap from detailing the contents of the blog folder or just update it every so often with the contents of the blog folder?
Intermediate & Advanced SEO | | PeterAlexLeigh0 -
Where to get a video sitemap creator for Wordpress?
I have a website that is nearly all about videos and is based on Wordpress. Does anyone know of a way to create a video sitemap that updates automatically as I write a new post? The video files and other data are all stored in separate meta-post locations... So it needs to be able to grab them. Any help is appreciated.
Intermediate & Advanced SEO | | DojoGuy0 -
What's your best hidden SEO secret?
Don't take that question too serious but all answers are welcome 😉 Answer to all:
Intermediate & Advanced SEO | | petrakraft
"Gentlemen, I see you did you best - at least I hope so! But after all I suppose I am stuck here to go on reading the SEOmoz blog if I can't sqeeze more secrets from you!9