There's a good history of it in Lego here [0]. You can see its a complex relationship but yellow has been used for caucasians (and asians) and has always had a difficult relationship for people of color.

[0] https://www.quora.com/Why-do-LEGO-minifigures-have-yellow-he...