Host sitemaps on S3?
-
Hey guys,
I run a dynamic web service and I will start building static sitemaps for it pretty soon. The fact that my app lives in a multitude of servers doesn't make it easy to distribute frequently updated static files throughout the servers.
My idea was to host the files in AWS S3 and point my robots.txt sitemap directive there. I'll use a sitemap index so, every other sitemap will be hosted on S3 as well.
I could dynamically mirror the content from the files in S3 through my app, but that would be a little more resource intensive than just serving the static files from a common place.
Any ideas? Thanks!
-
My general take on this sort of scenario is first to eliminate all the redundant hostnames with round-robin DNS, through adding extra server power with software-based load-balancing in the interim with a solution like InterWorx, and breaking out database servers. If you do that, you should have a nice little server cluster that's crazy efficient.and scalable. You can add a CDN to the mix if you like as well. With all of that, SEO should work the same way as on a single server.
Sitemaps can then be generated dynamically really easily (in under 25 lines of code, most of the time).
If you just want a way to mirror static files, you'll want to look at rsync.
And finally, as for S3, my personal opinion is to stay away. I'm an SEO, but I also spent 7 years building a hosting company. Those solutions sound great in their marketing, but are scientifically less reliable than standard hosting, and you can verify that via public uptime tracking sites like HyperSpin.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Which product URL to include in Sitemaps?
Hi Does the product URL's in Sitemaps affect the sub-categories authority too? For example, if I have a product with 2 URL's and which have a canonical tag: **/brands/michael-kors/bags/**jet-set-double-zip-wallet/ **/women/accessories/wallets/**jet-set-double-zip-wallet/ If I make the main URL "/women/accessories/wallets/jet-set-double-zip-wallet/" and set that as the Canonical URL & list that URL in the XML Sitemap, will it also mean the "/women/accessories/wallets/" category will get more authority and increase it's power to rank? Thanks Frankie
Technical SEO | | Frankie-BTDublin0 -
XML Sitemap Creation
I am looking for a tool where I can add a list of URL's and output an XML sitemap. Ideally this would be Web based or work on the mac? Extra bonus if it handles video sitemaps. My alternative is XLS and a bunch of concatenates, but I'd rather something cleaner. It doesn't need to crawl the site. Thanks.
Technical SEO | | Jeff_Lucas0 -
Can the Hosting location of image files have a negative effect if 'off-site' such as on the devs own media server ?
Hi Can the Hosting location of image files have a negative effect if 'off-site' such as if they are on the developers own media server ? As opposed to on the actual websites server or file structure ? In the case i'm looking at the image files are hosted on a totally separate server (a media subdomain of the developers site server) from the subject sites dedicated server. Will engines still attribute the properties of files hosted in this manner to the main website (such as file name, alt attributes, etc etc) ? Or should they really be on the subject sites server own media folder ? Cheers Dan
Technical SEO | | Dan-Lawrence0 -
Satellite Website Dilemma - Hosted in House or Elsewhere? Blog or Actual Shop?
Hi All, I have recently noticed a LOT of websites appearing in some of the SERP (example http://goo.gl/UyHZp6) that have an exact match domain (or as near as) and are either thin blogs that have a splash of content, and then link back to Amazon, competitor A competitor B etc. or an online secondary shop. Getting tired of this I have purchased a couple of exact match domains of my own, but am unsure of the best way to tackle this with long term gains in mind. The exact match domains I have are the .co.uk and com versions of these: http://goo.gl/xrjY7Z http://goo.gl/Mg0XBl The ideal scenario for me would be to create the satellite website as a functioning shop specialising in just a small group of specialist products (10 - 12) from a subcategory of the main site. My main store has 1200 + items and this will make the user experience better as I feel as it will make navigation easier, allow for more information to be present without confusing things. It would also allow the customer to feel safe knowing they are buying from a specialist. However I have the following in mind: My ecommerce software open cart supports multi store from the same database, this is great and makes management massively easy. It would allow me to brand the satellite store up as specialist store yet manage all orders through one admin portal. The sites would have separate IP addresses, but I am worried about the site being on the same server as the main site, and sharing whois info etc. Would google think of this as spamming the results? There will be no shared content, and I do not intend to interlink the sites for fear of them looking like a link network. The other option is to take out some cheap hosting and start building content on a blog similar to this: http://goo.gl/sBB3wY I hate this however as it just seems spammy and as a consumer it annoys me when I find this. What are your thoughts on how to deal with this?
Technical SEO | | speedingorange0 -
Is there anywhere i can find a list of the best UK dedicated hosting companies
Hi, i am after finding a list of uk hosting companies that can offer dedicated hosting. I have been looking for months now for a UK hosting company. I need the following or something similar Intel Xeon E3-1220 4x3.1GHz TB 8MB Cache 500GBStorage8GBRAM10TBBandwidthif anyone can help with this then that would be great.
Technical SEO | | ClaireH-1848860 -
Would you move the site to a different host or change packages at a significant expense in order to eliminate the meta refresh
When I began working with a site (http://www.visix.com) , I discovered a number of hosting constraints that hampered some SEO related changes I wanted to make. A year later, the site was teetering on the 1st page for a particular keyword of choice and when the Panda & Penguin updates happened, the site got passed by 3M & Amazon, both much bigger sites. (was #11, now #13) Now I'm thinking I should try and use the homepage to rank for keyword "digital signage software", where originally I was making progress with an inner page. Now I am revisting the homepage meta refresh and need to decide if it is enough of an issue to warrant a hosting change. http://www.visix.com has a meta-refresh "0" seconds to http://www.visix.com/index.aspx I know sites can rank well with these, although I don't know the level of handicap that it has. In an article here, http://www.seomoz.org/learn-seo/redirection there is a statement saying that a meta-refresh will not pass as much link juice as a 301 redirect. I have read about every opinion I can find, and would appreciate other's opinions on the matter. The host is Network Solutions and the hosting package does not allow 301 redirects, among other things. Would you move the site to a different host or change packages at a significant expense in order to eliminate the meta refresh or is it not a big deal on a well established site? Thanks very much for your feedback!
Technical SEO | | IntegralOCR30 -
Google rankings dropped dramatically after 24 hrs of hosting suspension
Hi, One of my websites ( http://www.traveldestinationsearch.com/ ) dropped most of its Google rankings after 24 hours of hosting suspension (from April 26 until April 27, 2012). The hosting company suspended my website after exceeding the bandwidth limit: there was no unusual activity on my website, it just exceeded its bandwidth limit by 20-30MB for the previous month. Anyway, the website is back online since April 27 but the problem is that, following these 24 hrs of no service, I see a dramatic decrease of my website's Google rankings for its main keywords. Even today, April 29, I can't find my website anywhere in the first 100 results for most of its targeted keywords. Before the suspension, the website ranked #1 for its main keyword and somewhere in the first 2-3 pages of Google search results for other two main keywords. My question is: is it the hosting suspension the reason for the Google ranking drop, and (assuming this is a temporary problem) when do you think I should expect my website to regain the rankings it had before the hosting suspension? Thanks for your support. Regards, Adrian
Technical SEO | | AdrianBanu0 -
How to generate a visual sitemap using sitemap.xml
Are there any tools (online preferably) which will take a sitemap.xml file and generate a visual site map? Seems like an obvious thing to do, but can't find any simple tools for this?
Technical SEO | | k3nn3dy30