Re:fMRI-to-Image with Contrastive Learning and Diffusion Priors
by theaiquestion on Tuesday July 18, 2023 @12:26AM
(#36767800)
I think the method of merging the pipelines via img2img should be replaced with controlnet. Possibly needing to be finetuned specifically for this, although existing controlnet models might work fine for this.
This is exactly what you'd want to use controlnet for - mapping semantic information onto the perceived structure.
Re:fMRI-to-Image with Contrastive Learning and Diffusion Priors
by tmabraham on Tuesday July 18, 2023 @02:56AM
(#36768810)
Yes we've been looking into ControlNet as well, and I think there is one recent fMRI-to-image paper that also has tried ControlNet. Maybe we'll use ControlNet in MindEye v2 :)
This is exactly what you'd want to use controlnet for - mapping semantic information onto the perceived structure.