I remember a similar New Yorker article ~5 years ago about medical imaging ML/AI where they realized it's good hit rate was actually a data artifact from training. Something along the lines that essentially all the positives had secondary scans and so there was a set of known positives which had say run through the same machine/lab and had similar label color & text markings in the margin of the imaging.

When they went back and tested with clean images that didn't basically have the "im a positive cuz I have this label over here in the margins", the hit rate dropped below that of humans.

It was an article with anecdotes about some of the hospice cats that seemingly are able to detect when a patient is about to die. Entirely possible as they have a sense of smell and patient tumors likely giving out detectable odors.

Nonetheless, the ML model & the cat were similarly inscrutable.