Of course, because it's discrimination
The reason is relevant because it results in discrimination. If someone doesn't want to interact with me because I'm being an asshole, that's perfectly OK, if someone doesn't want to interact with me because I have white skin, that's discrimination and that's not OK
Not if their choices result in discrimination
The government is not "an uninvolved third party". The civil rights act of 1964 guarantees the protection against discrimination based on race, color, religion, sex or national origin. The gov.'s responsibility is enforcing the law.
That protects one parties freedom to not be discriminated against, like I said before, nobody has the right to discriminate against people based on what I listed