Considering that the Indian immigrant proprietor of the convenience store in the show is actually brown, I'm confident that the vast majority of people considered the yellow-colored Simpsons to be a white family.