I think the veil of ignorance is a great thought experiment everyone should think about, but there shouldn't be any prescribed way how to think about it.
Stated another way, while that argument is probably a valid way to "win" a competitive debate, it doesn't strike me as particularly useful for exploring the concept that Rawls is presenting.
Abstractly, Rawls says society shouldn't be racially/ethnically/socioeconomically/etc prejudiced. The vast majority of people agree with this obviously.
Concretely, he says this can be achieved by ignoring your racial/ethnic/socioeconomic/etc status.
The claim is that the more you ignore your own lived experiences, the less you can use the knowledge that it gives you.
Rawls also argued that people would maximize the minimum utility behind the veil of ignorance, as they wouldn't want to risk being a persecuted minority.
This didn't come up during the debate, although it should've since it's why "whatever rules society operates under need to work regardless of what your starting point in society is".
There's a strong argument that the knowledge someone has from being disadvantaged contributes to their views on justice. The original position also incentivizes ignoring the rights of small groups of people since you likely aren't going to be a part of one.
One also gets into issues like reparations for past injustices. Those benefit a small group of people at the cost of the many (hard to justify with the original position), but can still be a form of justice since we want to balance past wrongs.