I think it's a class/culture thing, not a skin color thing. Hispanic implies you speak Spanish at home, likely to be an immigrant, maybe 1st or 2nd generation to be born in America, and your food/holiday/overall culture is different from the old "white" American one depicted in old TV shows and movies.