> It’s perfectly reasonable to think X people dying is worth it to avoid Y people’s quality of life being reduced by Z, depending on the values of X, Y, and Z.

not if I'm part of the X group, for purely selfish reasons, and it's perfectly rational for anyone to not want to be in the X group, no matter what values assigned to Y and Z are.

So the crux of the problem isn't what you stated - it's that these decisions are made, and some people disagree, because either they are in the X group, or not. This is why conscription is required in a war, and this is why lockdowns are needed in a pandemic, etc.