Yes, Google has a number of penalties that actively hurt or de-indexes websites with shady links.
Just focus on getting backlinks that would be useful to people actually visiting those websites. If a human found your link, would they respect you for it? That's the question you should ask when you build links.