I would also argue that a lot of times people are called racist (or called out for saying racist things) these days are likely because people are also less likely to give others the benefit of a doubt for their actions or words. This is likely complicated in that it appears some people that do harbor racist views feel more emboldened to speak them, but I'm not sure how to pick apart how much is one and how much is the other.
Since you're running with the candy analogy... your logical next step is way off. GPs candy analogy was supposed to be simple such that we could all start on the same page - either you're being purposefully difficult to try and argue your racism point, or are simply missing it. I like all colors of starburst. They taste different. Different is GOOD. I don't want an all orange package.
I think that's easily proven false. I value diversity of ethnicity and race because I think it's statistically more likely that it results in diversity of culture, and I think that's the real useful thing to be aiming for.
In the United States "white" isn't a race any more than "black" is. It's a short-hand for a cultural identity that our society foists upon you, and it comes with a mostly shared set of experiences and understanding.
A "black" person (which could be historically from Africa, Asia, South America or elsewhere) will likely face some shared set of experiences, as will a "white" person, or an "asian" person, or a "latino" person. The same could be said for a U.S. citizen or a European.
If you think "white" was used as shorthand for "the largest and most dominant cultural identity in America" as I do, then you think that diversity in that brings different perspectives and cultural norms, and some people find that refreshing and interesting (as well as there being some evidence that it provides many other benefits and detriments to various degrees).
So, do I think skin color is an important determinant of some quality in this discussion? Yes. In the case where you're looking at a mostly homogeneous group based on age and likely cultural identity, I think encouraging differences in both age and that cultural identity are likely to introduce more cultural diversity than some other options, and that's based on sociological and statistical beliefs. You may or may not think that qualifies as racism, but I don't, and part of that is because I don't think the result is based on actual skin color, just skin color in a geographic region (that is, a country with a different cultural and ethnic makeup may not see similar results). That means skin color is little other than a statistical marker for other, more useful attributes in this case.
The important thing being conveyed is lack of diversity, not whiteness. In the United States, saying something is "mostly white people" conveys that other ethnic groups are not present for some reason (whether it be them self-selecting to not participate or because of exclusion). The purpose is almost always to note diversity or lack-thereof, and assume that the person is making a value judgement about the people because their skin is white is to completely miss the point.
For example, if you bought a pack of starbust candies, and they were almost all orange, you might note or complain that they were almost all orange. I would do so, and orange is my favorite flavor of those. The issue is, I desire the diversity of colors and flavors, and too much orange is not as good as what the added variation brings.
So, "mostly old white people" as a negative doesn't mean old people are bad, or white people are bad, or old white people are bad, but that mostly old white people are not as good as a group that has more variation in it. "Mostly" is the word you should be focusing on, not the other ones.