It is often very hard to tell the difference between Behavior X is intrinsically harmful and Behavior X is mostly practiced by horrible people who are generally harmful to those around them. 

Stigma and taboo can drive good, upright, pro-social people away from damn near anything; good, upright, pro-social people often don’t want to risk hurting or upsetting others, and even if they’re not genuinely altruistic in that particular way, they generally have reputations to lose.  Which means that, once the stigma or taboo is firmly in place, you’re left with a core of deviants who can’t control themselves and/or don’t care about being labeled as scumbags because they’re already scumbags.  And, surprise surprise, everything those people touch turns to shit, they make their friends and relations miserable, so you’re left with a pattern of taboo-breaking correlating strongly with awful outcomes. 

Or maybe the taboo really is important, because the proscribed behavior really does yield awful outcomes!  That can sure be a thing.  But how can you know?  Are you going to conduct a study?  Of course you’re not, that would require breaking the taboo over and over in the name of social science.  And there are seldom a lot of good natural experiments to observe, because of the aforementioned social sorting.  In the end, your ability to discern the truth of the matter is very limited. 

But it’s worth trying to imagine the taboo being broken by the most careful, kindhearted, well-meaning people you know, and asking – would that be such a problem?