Best server-side sitemap generators
-
I've been looking into sitemap generators recently and have got a good knowledge of what creating a sitemap for a small website of below 500 URLs involves. I have successfully generated a sitemap for a very small site, but I’m trying to work out the best way of crawling a large site with millions of URLs.
I’ve decided that the best way to crawl such a large number of URLs is to use a server side sitemap, but this is an area that doesn’t seem to be covered in detail on SEO blogs / forums. Could anyone recommend a good server side sitemap generator? What do you think of the automated offerings from Google and Bing? I’ve found a list of server side sitemap generators from Google, but I can’t see any way to choose between them. I realise that a lot will depend on the type of technologies we use server side, but I'm afraid that I don't know them at this time.
-
Unless they have fixed it in recent months, xml-sitemaps does not generate correct video sitemaps.
-
Yeah, they offer free and paid hosted versions too. But I found the server side version much simpler to setup and control.
-
-
Excellent advice Federico. My first reaction was, "but that's not a server-side sitemap generator". I just looked at their website though and it turns out that it is! Looks like I need to read things more carefully!
I'll look into that as an option but if anyone else has any server side sitemap generators that they'd recommend then I'd be really interested to hear about them
-
I have been using xml-sitemaps (paid version) for all my sites over 5 years and they work like a charm, scraping and indexing what it needs to be indexed ans scraped, plus it consumes really low resources. 100% recommended (they have nice plugins too for extra sitempas (video, news, images, etc).
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is the best way to leverage Fiverr for content marketing?
I am currently using Fiver for cleaning up grammar, building backlinks and summarizing topics and articles. Does anyone else have suggestions on other ways to leverage the Fiverr community or any other similar services?
Intermediate & Advanced SEO | | clickshift0 -
Servers matter for different domains for same keywords?
We have 3 domains in the same industry, but unique content on each site. The whois info is private, but the domains are all under the same owner registrant info. In terms of rankings, does it matter if we move them all into the same RackSpace cloud account? Will Google know the relationship somehow and if they do, will they care?
Intermediate & Advanced SEO | | TheDude0 -
Exact Syntax for Canonical to PDFs for Windows Server
Hi There, I have got in my web several PDFs with the same content of the HTML version. Thus I need to set up a canonical for each of them in order to avoid duplicate content. In particular, I need to know how to write the exact syntax for the windows server (web.config) in order to implement the canonical to PDF. I surfed the web but it seems I cannot find this piece of info anywhere Thanks a lot!!
Intermediate & Advanced SEO | | Midleton0 -
Best software or online tool to create Videos
Kindly let me know best software or online tool to create videos for website and youtube. I am also looking for animation videos etc.
Intermediate & Advanced SEO | | AlexanderWhite0 -
Best practice for expandable content
We are in the middle of having new pages added to our website. On our website we will have a information section containing various details about a product, this information will be several paragraphs long. we were wanting to show the first paragraph and have a read more button to show the rest of the content that is hidden. Whats googles view on this, is this bad for seo?
Intermediate & Advanced SEO | | Alexogilvie0 -
Sitemap on a Subdomain
Hi, For various reasons I placed my sitemaps on a subdomain where I keep images and other large files (static.example.com). I then submitted this to Google as a separate site in Webmaster tools. Is this a problem? All of the URLs are for the actual site (www.example.com), the only issue on my end is not being able to look at it all at the same time. But I'm wondering if this would cause any problems on Google's end.
Intermediate & Advanced SEO | | enotes0 -
How to auto generate a unique meta description?
The site I am working on is a code nightmare for starters. I'm editing a file called layout that controls the section of each page. The programmer from a while back got unique titles by putting this piece of code in: <title><?= $this->metaTag ?></title> In all the different controllers and stuff I can see where the title is the name of the product plus review or something to that effect. How do I do this for the meta description? Right now the meta description is static in the layout file, and so every page has an identical one. I was hoping there was a way to make the meta description automatically use the first 140 characters on the page or something. Something like this:
Intermediate & Advanced SEO | | DanDeceuster0 -
What is the best approach to a keyword that has multiple abbreviations?
I have a site for which the primary keyword has multiple abbreviations. The site is for the computer game "Football Manager", each iteration is often referred to as FM2012, FM12 or Football Manager 2012, the first two can also be used with or without spaces inbetween. While this is only 3 keywords to target, it means that every key phrase such as "FM2012 Tactics", must also be targeted in 3 ways. Is there a recommended approach to make sure that all 3 are targeted? At present I use the full title "Football Manager" in the the title and try to use the shorter abbreviations in the page, I also make sure the title tags always have an alternative e.g FM2012 Tactics Two specific questions as well as general tips: Does the <abbr>HTML tag help very much?</abbr> Are results likely to differ much for searches for "FM 2012" and "FM2012" i.e. without the space.
Intermediate & Advanced SEO | | freezedriedmedia1