atom_101
Re:fMRI-to-Image with Contrastive Learning and Diffusion Priors
on Tuesday July 18, 2023 @03:23AM (#36769008)
This is mostly correct, except that there is only one model. This model takes an fMRI and predicts 2 outputs. The first is specialized for retrieval and the second can be fed into a diffusion model to reconstruct images.
You can see the comparison in performance between LAION-5B retrieval and actual reconstructions in the paper. When retrieving from a large enough database like LAION-5B, we can get images that are quite similar to the seen images in terms of high level content, but not so similar in low-level details (relative position of objects, colors, texture, etc). Reconstruction with diffusion models does much better in terms of low-level metrics.
Re:fMRI-to-Image with Contrastive Learning and Diffusion Priors
on Tuesday July 18, 2023 @03:15AM (#36768949)
We did have a face reconstruction project planned. It is on the back-burner for now. That one will be based on something like the Celeb-A dataset instead of the Natural Scenes Dataset (images from MS-COCO) used here.
Re:fMRI-to-Image with Contrastive Learning and Diffusion Priors
on Tuesday July 18, 2023 @03:10AM (#36768906)
Yes controlnet will be used in the next version. For this one we couldn't get it working in time.