"I've never been black and can't really tell first hand" I take that as a disagree.
You aren't unusual. Most white people in the US aren't objective about their advantages. 53% or white women and 58% of white men don't see racism as something they benefit from and other experience difficulty from.
I'll just pull one statistic up, that of wages:
Over time, through recession and booms, white men out earn Black and Hispanic Americans by almost 25%. That's a heavy advantage.
What to do? Just some ideas. I know you are going to hate this.
Step up affirmative action policies.
Stronger enforcement actions with stiffer penalties for wage discrimination
Improved education opportunities for minorities and women
Financial assistance to minority owned businesses such as low interest government loans.
In other words, start giving real advantages to make up for systemic advantages that white men naturally have.