Imagine in your town, other people could force you associate with them, against your will, and force you to serve them, against your will.
If everyone could decide the same thing, I would hope that decision would include the equal right to peaceful self determination.
I don't have the right to force anybody to associate with me, if you're saying I do, how about getting busy cooking me supper and put a bottle of wine in the fridge for me? Oh and fluff up the pillows on the couch, that's where you're sleeping tonight.