> Humans can tend towards liking more highly saturated colours, which would be a distortion of the original image.
android with google photos did/does this whereas apple went with enhanced contrast.
as far as i can tell, they're both wrong but one mostly notices the 'distortion' if used to the other.
The next sentence says "The test subject is able to flip between the two distortions, and has the original image available on the side for comparison at all times.", which indicates that the subjects weren't shown only the distortions.
The change was literally just made: https://github.com/google-research/google-research/commit/4a...
It appears this was in response to Hacker News comments.
One other thing to control for is the subpixel layout of their display which is almost always forgotten in these studies.
Looking further [1]:
> It consists in requiring a choice between two different distortions of the same image, and computes an Elo ranking (an estimate of the probability of each method being considered higher quality by the raters) of distortions based on that. Compared to traditional Opinion Score methods, it avoids requiring test subjects to calibrate their scores.
This seems like a bad way to evaluate image quality. Humans can tend towards liking more highly saturated colours, which would be a distortion of the original image. If it was just a simple kernel that turned any image into a GIF cartoon, and then I had it rated by cartoon enthusiasts, I'm sure I could prove GIF is better than JPEG.
I think that to produce something more fair, it would need to be "Given the following raw image, which of the following two images appears to better represent the above image?" The allowed answers should be "A", "B" and "unsure".
ELO would likely be less appropriate. I would also like to see an analysis regarding which images were most influential in deciding which approach is better and why. Is it colour related, artefact related, information frequency related? I'm sure they could gain some deeper insight into why one method is favoured over the other.
[1] https://github.com/google-research/google-research/blob/mast...