The term "collective good" is arbitrary and flexible and often oxymoronic. It was once believed by a majority of voters that the "collective good" could be served by prohibition wasn't it?
If I, as an individual, should have no power to run your life for you or determine how much of your money to take from you under threat of force, how can a group of people have that right? They shouldn't.
Logic insists that if you have the right to repel individuals making threats against you or what you own, you would also have the right to repel a group or a gang (society, government, etc.) doing the same thing.
A society based in voluntary human interactions is infinitely more peaceful than one that is not. The collective good, if there even is such a thing, would be best served by leaving others alone rather than making their choices for them under threats for noncompliance.
The problem is the means used. You can't base an action in a threat and then point ONLY to the end result, all the while ignoring the means.