That's a serious issue. I play a stringed instrument, and purchased a "professional" model a few years ago. Among the instruments that I was trying out, some of the physical differences were sufficient that there would have been little expectation of identical sound, so the game was to discover the differences and then make a choice.

There were instruments that I rejected within seconds, but others that took me upwards of 1/2 hour to figure out how to adapt my playing to the characteristics of the instrument. There were issues that might have been unnoticeable to an audience member, but that would have affected my ability to play with good technique over the duration of a performance. Some of the instruments, including the one I ended up buying, changed my opinion of how I wanted to develop my tone and playing style.

Granted, I was always faced with a slight doubt about whether any of it was real, but without any good way of settling that issue. Also, a virtuoso might have been able to make all of the instruments sound the same with much less effort.

Oh, and don't get me started about bows.

They also mention in the post that the old violins were shipped, and no adjustments made, which means bridges could be improperly positioned, sound posts in poor positions... having these properly adjusted can make the difference between a great sounding instrument and a screech.

I took the Pepsi Challenge at a trade fair in the 70's, and had no trouble distinguishing the Coke. I preferred the Coke, said so, and the Pepsi spokesman's response was:

  "Next!"

I figured I wasn't getting on camera.

I'm surprised that the Pepsi Challenge thing isn't more obvious to people. I mean, hasn't everyone experienced this themselves when it comes to those "healthier" versions of snack foods?

First bag: "Hey, these taste much better than I expected! They're just as good as my normal snack. Eating these instead of my old snacks will be easy."

Second bag: "They're pretty good, but I do taste the difference."

Third bag: "Oh god, get those away from me."

It's not just sip test versus can, but it's how things taste when you come back again and again.

Anecdata:

My friends and I will often dial in a guitar or bass tone that is very pleasing when we play individually, but sounds like junk when you have guitar + guitar + bass + drums. After noticing this we also noticed that a lot of the bands we thought sounded the best used tones that were individually good but not fantastic because the were shaped to best fit the sonic space they existed within than to sound amazing in their own right.

You can find similar issues with loudspeaks (those overly enhancing basses sounds better, but perform worse) and TV displays (overly saturated colors).

Indeed, one of the challenges of comparing audio systems is that -- rule of thumb -- louder sounds better. To have any hope of exposing effects other than overall loudness, systems under test have to be matched for gain.

If it is to compare my thumb say that —at the same loud the better sounds lower.

I've had this same criticism of audio A/B tests: 30 second clips are just not decent to get a sense of the feeling a particular setup or recording imbues; only its quantitative momentarily detectable constructs. And only the ones the puny human brain and ear can discretely identify and remember, at that.