Re:fMRI-to-Image with Contrastive Learning and Diffusion Priors
by atom_101 on Tuesday July 18, 2023 @03:15AM
(#36768949)
We did have a face reconstruction project planned. It is on the back-burner for now. That one will be based on something like the Celeb-A dataset instead of the Natural Scenes Dataset (images from MS-COCO) used here.
Re:fMRI-to-Image with Contrastive Learning and Diffusion Priors
by tmabraham on Tuesday July 18, 2023 @04:35AM
(#36769520)
Yeah using data from a 7T MRI giving higher spatial resolution definitely helps!
The fMRI dataset includes signal from the whole brain but we only use the data from the visual cortex for this study.
Re:fMRI-to-Image with Contrastive Learning and Diffusion Priors
by StackOverlord on Tuesday July 18, 2023 @10:20AM
(#36772759)
Are you able to extract an image showing the screen in the fMRI machine, as the subject can see it in between pictures ?
One of the biggest issues with any attempt to extract information from an fMRI scan is resolution, both spatial and temporal - this study used 1.8mm voxels which is a TON of neurons (also recall that fMRIs scan blood flow, not neuron activity - we just count on those things being correlated). Temporally, fMRI sample frequency are often <1hz. I didn't see that they mentioned a specific frequency, but they showed images to the subject for 3 seconds at a time so I'd guess that's designed to ensure you get a least a frame or three while the subject is looking at the image. You can sort of trade voxel size for sample frequency - so you can get more voxels, or more samples, but not both. So detecting things that happen quickly (like, say, moving images or speech) would probably be quite hard (even if you could design an ai thingey that could do it, getting the raw data at the resolution you'd need is not currently possible with existing scanners)
Also, not all brain functions are as clearly localized as vision - the visual cortex areas in the back of the brain map pretty directly to certain kinds of visual stimulus, while other kinds of stimulus and activity are much less localize (there isn't a clear "lighting up" of an area). You can get better resolution if you only scan part of the brain (i.e. the visual cortex) (I don't know if that's what they did for this study), but that's obviously only useful for activity happening in a small part of the brain.
ANYWAY SO COOL!!! I wonder if you could use this to draw people's faces with a subject who is imagining looking at a face? fMRI police sketch? How do brains even work!?