Hi Alan,
I would remove the links. If you're indeed suffering from a penguin penalty, your site will suffer as long as you have those links. Ask these directories/sites to remove their links to you, or disavow them. Keep a log of your backlinks, the ones you've identified as spammy, and the actions you've taken to clean up your link profile.
I know you're concerned about removing these links and how it will affect your search rankings. As long as you're under a penalty, your site won't do as well in SERPs. Cleaning up your link profile will get you out of the Google doghouse, and help you on your way to better SERP listings. Make sure you're working to get links from highly authoritative websites, and you'll be in a better position.
Good luck!
Morgan