What's missing here is the risk of the medical device not performing as expected on real patients. These risks are usually mitigated reasonably by medical device manufacturers and designers such that it doesn't matter that "AI medical devices" are not trained on real patient data.

The FDA keeps a database of all adverse events reported by manufacturers or healthcare systems [1], so we can check in a few years if these AI medical devices are causing an uptick of complaints.

[1] https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfmaude/s...