That's how averages work unfortunately.

Conversation looks like this:

A: X is true.

B: Please provide proof.

A: Here is metric using averaging over large populations.

B: Using averages over large populations is hiding important details about X.

A: How else we could compare large populations?! We must use globals averages!

B: Averages have the effect of very small number of very bad instances distort the result of large population which would have much better results, thus making it appear bad results are much more common than they are actually are. We know specifically this is the case for the question we are discussing.

A: Well, that's how averages work, nothing can be done about it.

Don't you think something is missing here?

No, not really. If your job is to compare countries, how else would you do it? Seriously.

PS: you know medians exist and those studies account for it, right? Nobody is saying YOUR school is mediocre.

Aside from population-wide sums or “averages” (in the specific sense of “arithmetic mean”), there's also models, medians, various distributional measures, etc. Which subset (which may include more than one selection) of these are most appropriate for a particular comparison depends on the specific purpose of the comparison. Arithmetic mean and/or sums are usually the easiest measures, but quite often not the most relevant.

I know, I'm trying to understand what studies contradict PISA and why is it faulty.