The term "peanut gallery" originated from vaudeville theater, and comes from the fact that unruly viewers who were not happy with the performance would throw their concessions at the performers, mostly roasted peanuts as these were the cheapest concessions. The attendees of early vaudeville were primarily immigrants, who would now be considered white, but were not at the time (scottish, irish, italian, and german immigrants).
There's nothing particularly racist about the term, never has been, and still isn't as it's used in modern parlance. Whether or not it's acceptable to use phrases with classist or class-based origins in your speech and writing is an exercise I leave up to you.
I did share a bad article I didn't read. I will do some more research on peanut gallery specifically. In either case, I probably still won't use it since it has this connotation to some people.