Yellow doesn't represent anything, it represents nothing. It's a blank hole that people can fill in with their biases. White people will picture it to be white, black people can imagine it being black. That becomes a problem when you want to represent a black character, to a bunch of white people, who consider him being black an important part of his character. In other words it's (very deliberately) a bad tool for talking about race.