Best server-side sitemap generators
-
I've been looking into sitemap generators recently and have got a good knowledge of what creating a sitemap for a small website of below 500 URLs involves. I have successfully generated a sitemap for a very small site, but I’m trying to work out the best way of crawling a large site with millions of URLs.
I’ve decided that the best way to crawl such a large number of URLs is to use a server side sitemap, but this is an area that doesn’t seem to be covered in detail on SEO blogs / forums. Could anyone recommend a good server side sitemap generator? What do you think of the automated offerings from Google and Bing? I’ve found a list of server side sitemap generators from Google, but I can’t see any way to choose between them. I realise that a lot will depend on the type of technologies we use server side, but I'm afraid that I don't know them at this time.
-
Unless they have fixed it in recent months, xml-sitemaps does not generate correct video sitemaps.
-
Yeah, they offer free and paid hosted versions too. But I found the server side version much simpler to setup and control.
-
-
Excellent advice Federico. My first reaction was, "but that's not a server-side sitemap generator". I just looked at their website though and it turns out that it is! Looks like I need to read things more carefully!
I'll look into that as an option but if anyone else has any server side sitemap generators that they'd recommend then I'd be really interested to hear about them
-
I have been using xml-sitemaps (paid version) for all my sites over 5 years and they work like a charm, scraping and indexing what it needs to be indexed ans scraped, plus it consumes really low resources. 100% recommended (they have nice plugins too for extra sitempas (video, news, images, etc).
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Sitemaps: Best Practice
What should and what shouldn't go in the sitemap? In particular, pages like subscribe to our newsletter/ unsubscribe to our newsletter? Is there really any benefit in highlighting those pages to the SEs? Thanks for any advice/ anecdotes 🙂
Intermediate & Advanced SEO | | Fubra0 -
To include in Sitemap or not to include?
Hello all, A bit of a confusing one but please bear with me... On our website we have a Used Cars section where each morning a feed is loaded onto our site with any changes to the stock. Some cars may have been sold and removed, some new cars may be added, some prices may be changed, every day every morning this very large section of our website is updated. The question I have is, should I be including these urls in my sitemap? The Used Cars section is a huge portion of our website content and is our most important area, the Used Cars overview is our most frequently visited page. The reason I ask is because of course Google might crawl and see car X, but tomorrow car X could be gone and be replaced with car Y. Should I be even mentioning these pages to Google if by tomorrow some of those urls could be gone? It's always changing and it's something we don't have control of. Thanks!
Intermediate & Advanced SEO | | HB170 -
How Should We Best List Events Pages?
Hi everyone! Luke here from CHARGED.fm hoping that a brilliant mind could help me with another annoying (at least for me) technical seo question. It's about how we list the events on our ticketing site. Here's the rundown: We currently list tickets by event id, but our competitors keep the event page in the same silo and use the venue name and date of event in the url. So we do this: http://www.charged.fm/kinky-boots-tickets (disregard redirect for now) List the events where you can choose from these: http://www.charged.fm/event/tickets/2518362/kinky-boots
Intermediate & Advanced SEO | | keL.A.xT.o
http://www.charged.fm/event/tickets/2511448/kinky-boots Moz lists these as duplicate content, so we're wondering how to resolve this. We're also wondering if it would be benficial to keep the events page in the same silo like our competitors: http://www.vividseats.com/theatre/kinky-boots-tickets/kinky-boots-9-20-1537274.html (notice how they go /theatre/kinky-boots-tickets/event/) Would it be beneficial to list like this? Is it inconsequential? Could we leave things the way that they are or should we at least add the venue and date to the events page URL? Thanks a lot for any help,
Luke0 -
Best-of-the-web content: Graphical Tips
This question is for EGOL (if he's willing) and anyone else who wants to partake. EGOL is the best content writer I've ever run into, really. I'm wondering what his top 3 to 5 tips are on how to use graphical layout (font, images, graphics, organization, menu, etc) to make content irresistable. A couple of assumptions: The content is written really well from a perspective of authority. Also, we're not including video on this one. Again, anyone is welcome to answer this. Thanks!
Intermediate & Advanced SEO | | BobGW1 -
Sitemap Folders on Search Results
Hello! We are managing SEO campaign of a video website. We have an issue about sitemap folders. I have sitemaps like ** /xml/sitemap-name.xml .** But Google is indexing my /xml/ folder and also sitemaps and they appear in search results. If i will add Disallow: /xml/ to my robots.txt and remove /xml/ folder from webmaster tools, Google could see my sitemaps? or it ignores them? Will my site effect negatively after remove /xml/ folder completely from search results? What should i do?
Intermediate & Advanced SEO | | roipublic0 -
Domain Name Change - Best Practices?
Good day guys, We got a restaurant that is changing its name and domain. However they are keeping the same server location, same content and same pages (we are just changing the logo on the website). It just has to go a new domain. We don't want to lose the value of the current site, and we want to avoid any duplicate penalties. Could you please advise of the best practices of doing a domain name change? Thank you.
Intermediate & Advanced SEO | | Michael-Goode0 -
Changing Servers + Effect on SEO
Hi, I am currently with a very slow server. Our website takes quite a while to load, FTP is very slow and content changes with Wordpress are slow because even the database connection takes a lot of time. However, my website ranks very well. Traffic has doubled in the last year. Our domain has been registered with this company for over 10 years. I am wondering if changing to a different hosting provider would have an effect on my rankings due to the change in IP.
Intermediate & Advanced SEO | | MangoMan160 -
Best way to improve page rank
I notice many small business sites seems to have a page rank of 3,4, or 5 which don't appear to be doing a great deal of SEO on their websites. i.e these are very basic sites with a little static content that rarely changes, no blogs or particular links. Does having a high page rank still mean your will achieve better search engine positions? whats the best way to improve page rank for small business sites? thanks
Intermediate & Advanced SEO | | Bristolweb0