Maybe most just don't want to? In my small town, most black folks who take a job here move away eventually because there aren't many blacks living here, and they want to be among black folks. That does not make them racist, that is just their preference. Some stay, and they are treated just like everybody else.
We hire lots of engineers here. HR is under huge pressure to recruit minorities. A black, female electrical engineer commands about a 30% higher starting salary than a white, male EE. Black, female electrical engineers are quite a rarity despite that. In fact, I only know of one.
I would love to live next door to Michael Jordan but "they" won't let me. That could be racist, but I doubt it.