You could always chop off the final softmax layer and dump out the activations of one or two layers below, which are basically an embedding, which should reflect that it's partly cat and partly truck?

It might not be clear from the article, it took me a bit of scanning back and double checking myself: the impossible nonsense picture is _overlaid_ on the left hand picture to produce the right hand picture

which is, at least to our conscious verbalizing mind, ~indistinguishable from the left hand picture. the interesting part is they _are_ distinguishable.

as you point out, you'd expect a human to just be like "uhhhh...either?", but it turns out we do see something subconciously, because people do identify the overlaid image at > chance

I don't think it is subconscious. In the bottom left of the vase picture, the "cat-like" version has "whiskers". The egg has a sort of caty blob on the right (though it's hard for me to decide whether the cat or truck version is more pronounced). The street picture has a white blob in the center that's "sheepier" than the chair version.

To me it seems like a conscious "if I had to pick the more cat-like one, this picture has a bit right here that sort of looks cat-like".

i.e. it's weird to say the images "influence" us. It seems more accurate to say that when prompted, we can perceive/understand what is confusing the model.

In the case of giving people to look at subtly-distorted images and asking them "truck-vs-cat" and "chair-vs-sheep", I think it's likely people thought: "Huh, I guess one of those involves artificially constructed angles and the other tends to organic rounded bits, so I'll decide on which seems extra-present."

To me that seems a reasonable--and not very exciting--explanation for why humans may get similar this-or-that answers to a machine-vision model.

Yeah but that's not what happened