To call it "white advantage" suggests that the advantage is inherent to being white and not simply because of the current distribution of population.
White advantage or white privilege is not the about the WHY of the advantage. The why could be population, wealth concentration, power structures....any number of things or any number of combination of things. There is no inherent difference between the races, you are correct.
'white privilege' is just the name for the advantages (more dating choices, etc) that you and I have because we have white skin and would not have if we had dark skin, in this particular society.
Race is brought into it because it is a label for advantages that one race has over another race. that is what we are talking about, racial advantages/privileges in our society. It is just a label so we can talk about it.
There are other privileges, like male privilege. We can't leave the 'male' of the label because it is a label for the privileges males in this society have.