I remember a friend in an allied discipline talking to me about what they did for cervical cancer where again the initial AI outcomes are weak. I don't know if this is now gold standard clinical practice, or still an experimental, but here's how I remember it:

We screen people with a cervix for cancer by scraping away a tiny sample of the cells periodically and having that examined at a laboratory for anomalies. If caught early, cervical cancer isn't fun but it's extremely survivable. You don't technically need a cervix (after all about 50% of humans don't have one) so in the worst case hysterectomy (removal of the womb and cervix) is an option.

Machines aren't very good at looking at a slide full of cells from the human cervix and giving it a score like 1-5 where 1 is "Fine" and 5 is "Cancer". This is a standardised task that her (human) team do every day, and she works with other bodies across the continent to ensure they're all doing a roughly similar job by looking at each others examples and checking they get the same number, so this way a doctor in Paris and one in Birmingham should get interchangeable results despite using different labs to process the samples.

However, it turns out the machines are stellar at a related task. "Is this sample infected with HPV?". Humans do not find this task easy and historically this test wasn't done anyway.

So maybe a human would do the first thing, and, if it was a bit borderline, then they'd check the second thing. Cervical cancer is almost always caused by HPV, so if you don't have HPV then you almost certainly don't have cervical cancer. But since the machines are great at that second problem you can reverse things. The machine processes every sample for HPV and then a human only looks at the positive ones, rating those.