Best Practices for adding Dynamic URL's to XML Sitemap

[seekjobs]

Hi Guys,

I'm working on an ecommerce website with all the product pages using dynamic URL's (we also have a few static pages but there is no issue with them).

The products are updated on the site every couple of hours (because we sell out or the special offer expires) and as a result I keep seeing heaps of 404 errors in Google Webmaster tools and am trying to avoid this (if possible).

I have already created an XML sitemap for the static pages and am now looking at incorporating the dynamic product pages but am not sure what is the best approach.

The URL structure for the products are as follows:

http://www.xyz.com/products/product1-is-really-cool
http://www.xyz.com/products/product2-is-even-cooler
http://www.xyz.com/products/product3-is-the-coolest

Here are 2 approaches I was considering:

1. To just include the dynamic product URLS within the same sitemap as the static URLs using just the following http://www.xyz.com/products/ - This is so spiders have access to the folder the products are in and I don't have to create an automated sitemap for all product

OR

2. Create a separate automated sitemap that updates when ever a product is updated and include the change frequency to be hourly - This is so spiders always have as close to be up to date sitemap when they crawl the sitemap

I look forward to hearing your thoughts, opinions, suggestions and/or previous experiences with this.

Thanks heaps,

LW

[DeanAndrews]

Hi LW

I agree with Mark re archiving products. Although our products don't expire as quickly as yours appear to do I use http://www.xml-sitemaps.com/standalone-google-sitemap-generator.html on a cron job to keep our sitemap fresh.

I also exclude some of our over dynamic URLs using this tool from appearing in the sitemap.

Dean

[Mark.RedGiant]

Hi LW,

What system is backing the online store? Are you using a CMS-driven e-commerce solution?

My suggestion would be to create an automated sitemap for the products. Pay careful attention to the priorities you assign and the update frequencies. (Hourly/daily is fine) I definitely think that you'd be spending far too much time on updating a sitemap if you had to do it manually.

This method will result in you having a more accurate sitemap on crawling.

Also, if you are planning on offering the same project in future, it might be an idea not to remove the product altogether, but rather have a page saying "This offer is currently not available" or something along those lines.

Another option might be to have an archive category of products, where all your expired offers can be placed, not available for order. This could allow you to keep your indexed pages, avoid 404s as well as use the product pages to direct new visitors to related/newer products should they see the products in the archive.

Just thinking out loud.

I'd be interested to see the website and the solution that you do eventually implement.

Regards
Mark