We don't even have the images needed, especially for unsupervised learning.
A chest x-ray isn't going to do the model much good to interpret a prostate MRI.
Add in heterogeneity in image acquisition, sequence labelling, regional and site-specific disease prevalence, changes in imaging interpretation and most importantly class imbalance (something like >90% of imaging studies are normal) it is really really hard to come up with a reasonably high quality dataset with enough cases (from personal experience trying).
With respects to training a model, IRB/REB (ethics) boards can grant approval for this kind of work without needing individual patient consent.
A chest x-ray isn't going to do the model much good to interpret a prostate MRI.
Add in heterogeneity in image acquisition, sequence labelling, regional and site-specific disease prevalence, changes in imaging interpretation and most importantly class imbalance (something like >90% of imaging studies are normal) it is really really hard to come up with a reasonably high quality dataset with enough cases (from personal experience trying).
With respects to training a model, IRB/REB (ethics) boards can grant approval for this kind of work without needing individual patient consent.