News for hackers. Stuff that matters.

boh423

Yes, the underlying NSD dataset used in the three prominent recent papers on this topic (including the one linked here) is problematic as it invites this (classification/identification, not reconstruction): It only has 80 categories. It has not been recorded with reconstruction in mind.

Reconstruction is the primary and difficult aim, but is what you want and expect when people talk such „mind reading”. Classifying something on brain activity has long been solved and is not difficult, it is almost trivial with modern data sizes and quality. At 80 categories and with data from higher visual areas (as in all modern public datasets) you could even use an SVM for the basic classifier and then some method for getting a similar blob shape from the activity (V1-V3 are map-like), and get good results.

If you are ignorant about the question whether you are just doing classification you can easily get too-good-to-be-true results. With these newer methods relying on pretrained features this classification case can hide deep inside the model too, and can easily be missed.

The community is currently discussing to what extent this applies to these newer papers (start with original post), self-correcting you could say: https://twitter.com/ykamit/status/1677872648590864385?s=20

(Kamitani has been working on the reconstruction question for long time and knows all these traps quite well.)

The deeprecon dataset proposed as an alternative here has out of distribution „abstract“ images and no class overlap between train and test images, so it’s quite suitable for proving that it’s actually reconstruction. But it’s also one order of magnitude smaller than the NSD data used for the newer reconstruction studies.