I have had interesting results during the farmer updates, I was originally effect by a downturn of about 40% in traffic, but after about 4 weeks the traffic came back.   In the Panda 2.2 update my page rank slipped from 6 to 5   But I did not loose any traffic despite loosing the page rank.

Have you considered using a CDN? This will cache your site in multiple locations so end users site experience will be better.

I had the most horrible experience with Joomla, Joomla is why I went to wordpress.  I had continual hacking when my site was on Joomla.  Is anyone using frameworks such as genesis or thesis?