I took "simply falling down in the SERPs" to mean that it's preferable to an actual penalty. We are all rising or falling in the SERPs for every keyword every day according to what changes happen in and around our sites and in and around other people's sites.
Google is trying to use a bunch of signals to imply authority and quality and along the way they are bound to judge some well intentioned sites wrongly. Someone I know has a site full of original and useful content. But he also has a whole bunch of duplicate content pages lying around which he forgot about from months ago, and a long list of duplicated archive pages due to poor use of tags in WordPress. He's placed a large ad above the fold on every page and has a high code to content ratio on the page. He has poor but not non-existent social interaction on his site and his site as a home-made feel to it - it also looks dis-organised - which I tend to think prevents people from ever engaging with his stuff as they can't see the article for all the surrounding clutter. He uses GA so G knows about his bounce rate too. You can't fault him on his content, but in the past when he didn't know better he did some fake link building but he doesn't do it any more.
He lost 75% of his traffic on 24th April. At least if this a result of "simply falling down in the SERPs", he can simply climb back up with a few changes. Except for the bad links.
I'm not sure what anyone can do about those, but then I am (reliably) told Penguin had nothing to do with inbound links.
Liz