What if an image is so contrived that the correctly linearized data averages into a flat, gray signal?

Won't the "correct" scaler succumb to the problem?

The answer is no. Here is why: such a contrived image will actually look gray, and so when it scales to a gray square, that is arguably correct.

We can see this with the Dalai Lama readily. If apply even a modest gamma curve to it to bring up its midtones, it turns more or less gray.