If you want a coherent understanding of anyone’s public policy position – really, anyone’s at all – you need to be able to process the idea that there are separate layers of value judgment that come with separate kinds of censure. 

It is not a contradiction to say “every person should have the right to exclude absolutely anyone from his [property / social group / bed] for any reason at all,” and also to say “it was wrong of you to exclude Person X and you shouldn’t have done it.”   

It is not a contradiction to say “every person should have the right to say anything at all,” and also to say “it was wrong of you to say Thing X, delete your post.”

Hell – it’s not a contradiction to say “every person should have the right to construct and maintain his own property however he sees fit” and also to say “it is wrong of you to build an ugly house that makes people sad by destroying the aesthetic character of the surroundings, you should feel obligated to redo it.”

You can call it consequentialist-axiology-versus-deontic-morality, although in fact that’s only one of many different metaethical schemata that can account for the thing I’m talking about here.  Freedom-of-action is important in all sorts of contexts, so much so that we make rules enshrining it, but even so, there are many useful standards by which actions can be judged to be bad / less-desirable-than-alternatives / whatever.  “It’s not wrong to do any of X or Y or Z” doesn’t negate “…but the world would be better if you did X and not either of those other things.” 

Someone criticizing your speech is not thereby attacking freedom of speech.  Someone criticizing your associations is not thereby freedom of association.  You get the idea.