In America it seems to have more to do with culture than ancestry. I would bet that there are lots of people that think of themselves as white that came over on slavers ships.

Of course if you could trace ancestry back far enough we are all black.

The South Africans used to have a colour chart.