There's a strong argument that the knowledge someone has from being disadvantaged contributes to their views on justice. The original position also incentivizes ignoring the rights of small groups of people since you likely aren't going to be a part of one.
One also gets into issues like reparations for past injustices. Those benefit a small group of people at the cost of the many (hard to justify with the original position), but can still be a form of justice since we want to balance past wrongs.
I think the veil of ignorance is a great thought experiment everyone should think about, but there shouldn't be any prescribed way how to think about it.
Stated another way, while that argument is probably a valid way to "win" a competitive debate, it doesn't strike me as particularly useful for exploring the concept that Rawls is presenting.
Abstractly, Rawls says society shouldn't be racially/ethnically/socioeconomically/etc prejudiced. The vast majority of people agree with this obviously.
Concretely, he says this can be achieved by ignoring your racial/ethnic/socioeconomic/etc status.
The claim is that the more you ignore your own lived experiences, the less you can use the knowledge that it gives you.
Rawls also argued that people would maximize the minimum utility behind the veil of ignorance, as they wouldn't want to risk being a persecuted minority.
This didn't come up during the debate, although it should've since it's why "whatever rules society operates under need to work regardless of what your starting point in society is".
It is easy to agree with it if you are talking about some racial minority.
Treating murderers or child rapists with the compassion you would want in their shoes is beyond most people.
It also has a number of very easy criticisms that can be set against it
You always have to consider all sides of every decision as well as higher order effects.
I think the numbers argument is a pretty different one than the original thought experiment.
Furthermore the argument that it is a bad idea isn't even strong. The veil of ignorance is powerful and interesting but voting is about different cultures negotiating to make decisions. That all but guarantees there will be different conceptions about what is fair that are not the veil of ignorance and those need to be respected. They aren't sociopaths, they just don't agree with you.
Not all thoughtful people come to the same conclusions, and it's nice to try to axiomatise these things anyway, so you have something to point at when trying to formulate new rules.
For instance, non-toddlers understand the Golden Rule, which seems good enough on its surface, but on deeper reflection can let bad behaviour through.
Hypothetically, I like bribes: I can pay others to change the rules in my favour. I also like receiving money for changing rules. So this passes the Golden Rule, but fails Kant's Categorical Imperative.
Isn't this like table stakes for being a thoughtful person who isn't a sociopath? The most influential thought experiment in contemporary philosophy is to not act like a toddler?