Best server-side sitemap generators
-
I've been looking into sitemap generators recently and have got a good knowledge of what creating a sitemap for a small website of below 500 URLs involves. I have successfully generated a sitemap for a very small site, but I’m trying to work out the best way of crawling a large site with millions of URLs.
I’ve decided that the best way to crawl such a large number of URLs is to use a server side sitemap, but this is an area that doesn’t seem to be covered in detail on SEO blogs / forums. Could anyone recommend a good server side sitemap generator? What do you think of the automated offerings from Google and Bing? I’ve found a list of server side sitemap generators from Google, but I can’t see any way to choose between them. I realise that a lot will depend on the type of technologies we use server side, but I'm afraid that I don't know them at this time.
-
Unless they have fixed it in recent months, xml-sitemaps does not generate correct video sitemaps.
-
Yeah, they offer free and paid hosted versions too. But I found the server side version much simpler to setup and control.
-
-
Excellent advice Federico. My first reaction was, "but that's not a server-side sitemap generator". I just looked at their website though and it turns out that it is! Looks like I need to read things more carefully!
I'll look into that as an option but if anyone else has any server side sitemap generators that they'd recommend then I'd be really interested to hear about them
-
I have been using xml-sitemaps (paid version) for all my sites over 5 years and they work like a charm, scraping and indexing what it needs to be indexed ans scraped, plus it consumes really low resources. 100% recommended (they have nice plugins too for extra sitempas (video, news, images, etc).
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Strategies for generating a knowledge panel for brand queries
Hello, I'm looking for any new ideas or experiences that might help in generating a knowledge panel for a brand query. I'm working with a company that has a strong brand, over 200K searches for the brand name alone each month, but they don't have a knowledge panel. When you search the brand name, the right side of the search results is just blank white space. Here are some of the things they do have, which tend to trigger a KP: Well built-out Wikipedia page Pages on Wikidata, Freebase, Crunchbase Home page organization schema Home page schema using sameas to connect the company to their social profiles, wikipedia page, and Freebase and Wikidata Hundreds of mentions in national press outlets Verified Twitter and Pinterest profiles (the same type of verification celebrities get) Generally, this all (and maybe even a lot less than this), would trigger a KP when you searched the brand name, but it just doesn't. No branded queries trigger a KP. It's interesting, because even the three smaller companies that they've acquired had (and still have) knowledge panels when you search those brands. So, I'm wondering if anyone has seen or worked through this behavior before? Or, even if not, any thoughts or ideas that we can do to nudge Google towards a KP for a branded search? Thanks!
Intermediate & Advanced SEO | | brianspatterson0 -
What is the best SEO way for a shop
Hi there ! A client want to sell some products on its future website but just a small range (the most part of this website will not be an online shop). The idea is to add a "shop" button in the menu to redirect clients in this shop. I would like your opinion about how should I construct this shop, what do you think is the best for SEO : "www.website.com/shop" or "shop.website.com" thank you in advance for your answers !
Intermediate & Advanced SEO | | EnjinFrance0 -
Best method for blocking a subdomain with duplicated content
Hello Moz Community Hoping somebody can assist. We have a subdomain, used by our CMS, which is being indexed by Google.
Intermediate & Advanced SEO | | KateWaite
http://www.naturalworldsafaris.com/
https://admin.naturalworldsafaris.com/ The page is the same so we can't add a no-index or no-follow.
I have both set up as separate properties in webmaster tools I understand the best method would be to update the robots.txt with a user disallow for the subdomain - but the robots text is only accessible on the main domain. http://www.naturalworldsafaris.com/robots.txt Will this work if we add the subdomain exclusion to this file? It means it won't be accessible on https://admin.naturalworldsafaris.com/robots.txt (where we can't create a file). Therefore won't be seen within that specific webmaster tools property. I've also asked the developer to add a password protection to the subdomain but this does not look possible. What approach would you recommend?0 -
Is a Dedicated Server a Worthwhile Investment?
Greeting MOZ Community: Our site is hosted on a virtual private server. Apparently there are dozens of other web sites hosted on the same server. The performance is usually pretty fast, with the site downloading in 1-3 seconds. However, a few times per month, the performance slows down, to say 5-6 seconds. Please see the attached image. I suspect this may have something to do with the other web sites on the server. Currently we pay about $60/month. A dedicated server would cost about $120/month. Would site performance be more consistent on a dedicated server? Could we enjoy potential SEO benefits by having our own server, i.e. could we rank slightly higher if the speed was more consistent and the performance slightly faster? Thanks, Alan 6D7kK61
Intermediate & Advanced SEO | | Kingalan10 -
Best option for Affiliate links on your website?
Hello! I have a website which is completely affiliate based. What is the best option for the links on-page? Examples would be: affiliate.website.com/12901730?2=3532523=user12342901730?2=3532523=user?Whittie www.website.com/affiliate=user?Whittie=load-of-tracking=date=blah=blaH?blah And So on... Which look ugly as sin when you hover over the Anchor Text. Ideally I would like a 301 redirect to mysite.com/goto/affiliatename, which would then have a rel nofollow. This way I could also track the exit pages via Analytics too guess, which I've not currently got set up and i'm desperate for it to be done. Does this method effect anything on search engines though? I've seen mixed report, but going back to 2011 which is too long ago in the SEO world. Another option is to use the likes of "Bit.ly" or use another domain and host 301s on there? The new bit.ly integration from moz might come in handy here. Please advise on the subject, I really appreciate any help on this, as i'm at a brick wall. Thanks
Intermediate & Advanced SEO | | Whittie0 -
Urls missing from product_cat sitemap
I'm using Yoast SEO plugin to generate XML sitemaps on my e-commerce site (woocommerce). I recently changed the category structure and now only 25 of about 75 product categories are included. Is there a way to manually include urls or what is the best way to have them all indexed in the sitemap?
Intermediate & Advanced SEO | | kisen0 -
Best url structure
I am making a new site for a company that services many cities. I was thinking a url structure like this, website.com/keyword1-keyword2-keyword3/cityname1-cityname2-cityname3-cityname4-cityname5. Will this be the best approach to optimize the site for the keyword plus 5 different cities ? as long as I keep the total url characters under the SeoMoz reccomended 115 characters ? Or would it be better to build separate pages for each city, trying to reword the main services to try to avoid dulpicate content.
Intermediate & Advanced SEO | | jlane90 -
Generating 404 Errors but the Pages Exist
Hey I have recently come across an issue with several of a sites urls being seen as a 404 by bots such as Xenu, SEOMoz, Google Web Tools etc. The funny thing is, the pages exist and display fine. This happens on many of the pages which use the Modx CMS, but the index is fine. The wordpress blog in /blog/ all works fine. The only thing I can think of is that I have a conflict in the htaccess, but troubleshooting this is difficult, any tool I have found online seem useless. Have tried to rollback to previous versions but still does not work. Anyone had any experience of similar issues? Many thanks K.
Intermediate & Advanced SEO | | Found0