> But I'm very curious how good one has to be in order to be better than a below-average doctor, or a 50th-percentile doctor, or a 75th.

In dermatology, on which I was working, models were better (at detecting skin cancers) than 52% of the GPs, going by just images. In a famous Nature paper by Esteva et al., the TPR was at 74% for detecting Melanomas. There is a catch which probably got underreported (The skin cancer positivity rate was strongly correlated to clinical markings in photos. Their models didn't do quite as well when 'clean', holdout data were used).

But the nature of information in all these models were skin deep (pun intended). They were designed with a calibrated objective in place unlike how we approach clinical diagnostics as open ended problems for the doctors.

Interesting, could you explain more about the clinical markings? Was this mentioned in the paper itself or was it later commentary?

I remember a similar New Yorker article ~5 years ago about medical imaging ML/AI where they realized it's good hit rate was actually a data artifact from training. Something along the lines that essentially all the positives had secondary scans and so there was a set of known positives which had say run through the same machine/lab and had similar label color & text markings in the margin of the imaging.

When they went back and tested with clean images that didn't basically have the "im a positive cuz I have this label over here in the margins", the hit rate dropped below that of humans.

It was an article with anecdotes about some of the hospice cats that seemingly are able to detect when a patient is about to die. Entirely possible as they have a sense of smell and patient tumors likely giving out detectable odors.

Nonetheless, the ML model & the cat were similarly inscrutable.

Nice, a model identifying the cases a doctor had enough doubts about to have a second test run.

Isn't it a tad unfair to compare a ML model for dermatology, only working with pictures, against general practitioners? IMHO comparing said model against dermatologists would be a better approach. And just working from images is not necessarily a dermatological model, buy rather an image analysis model.

> We test its performance against 21 board-certified dermatologists

From Andre Esteva's paper:

https://www.nature.com/articles/nature21056

Statistical mortality models are already more accurate than physicians for an average patient.