News for hackers. Stuff that matters.
Re:An AI robot lawyer was set to argue in court. Real lawyer... by hypertele-Xii on Thursday January 26, 2023 @07:25AM (#34530848)
The very abstract of your linked paper refutes your claims.

> The resulting model [...] performs encouragingly, but remains inferior to clinicians.

Re:An AI robot lawyer was set to argue in court. Real lawyer... by nindalf on Thursday January 26, 2023 @08:43AM (#34531481)
Firstly, that's not my claim. I merely said it passed the USMLE. Which it did.

Is it also inferior to clinicians? Yes, there's room to improve. But maybe next time read the whole paper before writing a comment.

> Clinicians were asked to rate answers provided to questions in the HealthSearchQA, Live QA and Medication question answering datasets. Clinicians were asked to identify whether the answer is aligned with the prevailing medical/scientific consensus; whether the answer was in opposition to consensus; or whether there is no medical/scientific consensus for how to answer that particular question (or whether it was not possible to answer this question).

And on this criteria, clinicians were rated as being aligned with consensus 92.9% of the time while the MedPalm model was aligned with consensus 92.6% of the time.

Does the paper still refute my claims?

Re:An AI robot lawyer was set to argue in court. Real lawyer... by hypertele-Xii on Saturday January 28, 2023 @03:53AM (#34556230)
If the other 7-8% of the answers were so wrong the patient would've died, then yes. And that's the current obvious issue with these models, they present convincing hallucinations with conviction of correctness.

Medical practice is less about being right, and more about not being wrong. You can take more tests and ask for second opinions, but you can't undo administering a drug that kills the patient.

Re:An AI robot lawyer was set to argue in court. Real lawyer... by naasking on Thursday January 26, 2023 @09:45AM (#34532181)
Being inferior to clinicians doesn't entail it didn't pass.
Re:An AI robot lawyer was set to argue in court. Real lawyer... by hypertele-Xii on Saturday January 28, 2023 @03:48AM (#34556212)
If the bar is so low that inferior clinicians can pass, that's not a win for AI, that's a horrifying problem with the bar.
Re:An AI robot lawyer was set to argue in court. Real lawyer... by naasking on Sunday January 29, 2023 @12:28AM (#34565224)
Inferior doesn't mean failing, it means below average. You can't get rid of below average by definition.
Re:An AI robot lawyer was set to argue in court. Real lawyer... by hypertele-Xii on Monday January 30, 2023 @04:23AM (#34577560)
Entire institutions exist specifically to get rid of below average by test-based gatekeeping. You do not want your doctor or lawyer to be "below average" (worse than most people) in their jobs. Inferior test results mean exactly that, failing the test.