You have the relationship backwards: being white meant no barriers. He never showed up for a meeting to be pointed at the back door or asked whether he was there to apply for the job as janitor or security guard. Nobody assumed he was there to take notes. He didn’t face constant questions about whether he could actually do something or was just “the diversity hire”.
That’s huge, especially when you look at the stats for things like loan or VC approval rates. Starting a company as a white man isn’t easy but it’s still easier than what most other people have to do.
Privilege mainly comes from wealth. Of course it is a serious problem in society when certain groups have disproportionate poverty or less wealth, because that indicates some structural cause.
However in the modern day that cause is simply the fact that most people, regardless of skin color, do not move out of their parents' income decile. And when one group is already more economically disenfranchised than other groups, then that economic disenfranchisement will perpetuate itself.
Try doing online dating as a White Guy and then doing it as an Asian guy: I can 100% assure you the response rate and experience is going to be wildly different.