Most doctors have insurance for covering their mistakes. We might expect an AI medical startup to pay analogous premiums when it’s paid analogous fees.

Exactly: skin in the game, and to underscore the point, make any debt non-dischargeable in bankruptcy.

Jesus they're calling us meat devs?

Reminds m of the assassin droid in KOTOR2 that called everyone meat bags.

We're getting there!

That's completely missing the point. The LLM score substantially higher than the clinician. Statistically this means the clinician will have many more misdiagnoses.

The point is that clinicians don't really get sued most of the time anyway for misdiagnoses. With AI, all one has to do is open up a new chat, tell the AI that its last diagnosis isn't really helping, and it will eagerly give an updated assessment. Compared to a clinician, the AI dramatically lowers the bar of iteratively working with it to help address an issue.

As for drug prescriptions, they are to be processed through an interactions checker anyway.

If you tell a LLM that its last effort was bad, it won't give you a better outcome. It will get worse at whatever you asked for.

The reason is simple. They are trained as plausibility engines. It's more plausible that a bad diagnostician gives you a worse outcome than a good one, and you have literally just prompted it that it's bad at diagnosis.

Sure, you might get another text completion. Will it be correct, actionable, reliable, safe? Even a stopped clock. Good luck rolling those dice with your health.

In summary, do not iterate with prompts for declining competence.

No, that's a gross frequentist assessment. In reality, the Bayesian assessment is contingent on the first response not helping, and is therefore more likely to be correct, not less. The second response is a conditional response that benefits from new information provided by the user. Accordingly, it's very possible that the LLM will suggest further diagnostic tests to sort out the situation. The same technique also works for code reviews, with stunning effect.

This recommendation isn't about prompts than include notes of "what didn't work". I'm talking about prompts that directly inform the model, "you are modelling an idiot".

The former is reasonable to include when iterating. The latter is a recipe for outcome degradation. GP above gave the latter form. That activates attention from parts of the model guiding towards confabulation and loss of faithfulness.

The model doesn't know what is true, only what is plausible to emit. The hypothesis that plausibility converges with scale towards truth and faithfulness remains very far from proven. Bear in mind that the training data includes large swatches of arbitrary text from the Internet, real life, and from fiction, which includes plenty of examples of people being wrong, stupid, incompetent, repetitive, whimsical, phony, capricious, manipulative, disingenuous, repetitive, argumentative, and mendacious. In the right context these are plausible human-like textual interactions, and the only things really holding it back from completion in such directions are careful training and the system prompt. Worst case scenario, perhaps the corpus included parliamentary proceedings from around the world. "Suppose you were an idiot. And suppose you were a member of Congress. But I repeat myself." - Mark Twain

The obvious next step is not that the LLMs replace doctors, it’s that LLMs become part of the ‘standard of care’, a component of the triage process. You go to the emergency room, and an LLM assessment becomes routine, if not required. This study shows that doing that will significantly increase accurate diagnoses for the start. Everyone wins.