Genuinely interested to know why - without being present - you think this would not have provided some sort of measurement of this. While it wasn’t scientifically devised it also wasn’t just “play a couple of things”.

The problem is that "what sounds better" isn't the same question as "which one is uncompressed".

The original CD version might have some high frequency stuff that's just on the edge of perception that you don't really know is there, but you can just sense a bit of discomfort when listening to it. After going through the MP3 process and that high frequency is removed because it contributes the least in reconstructing that signal, the resultant decompressed signal might sound "better" even though it's not the original, because you get a high quality reproduction but the thing that led to a slight discomfort when listening has now gone.

In this case, the sound engineer got them all right because he could tell the difference and knew what it was supposed to sound like. The rest of the people maybe could tell the difference or maybe couldn't (which was the claimed result of the test), but in fact, even if they could tell the difference, they had no idea which one was the uncompressed one and voted on which they thought sounded best.

As another comment has noted, it'd be a much better test if there was a "they sound the same" option as well as asking which one sounds best.

Playing A & B samples and asking which one is better/original requires much more from the listener that just hearing a difference between the two. It is possible to hear the difference, but not know which is which as that requires additional knowledge.

To avoid this issue you could:

Play (in random order) original twice and processed once and asking which one was different/processed.

Or play two sequences (in random order), [original, original] and [original, processed] and ask, if processed was in the first or second sequence.

Second option might focus better on short-term memory, because it has shorter sequences (2 samples vs 3 samples per sequence).

This would produce a better measurement of whether the difference is audible or not.

We actually did something more along the lines of your first suggestion - we were asking for difference not specifically “which is better” because actually “better” is entirely contextual (and actually this was part of the point - that some people actually prefer lower quality audio because it’s what they are used to). I think (it’s a long time ago now) that we played each set which was two or three samples twice and rather than asking which was “original” I think we asked them to score for clarity, richness and which they preferred - and then to make a guess about which was a higher/lower quality source file.

Like I say, it wasn’t a rigorously scientific experiment but it was in the context of a conference about evolution of audio standards and what that meant for audio delivery from labels/distributors to DSPs.