deepikaa_s

I built a multi-turn clinical safety eval framework for LLMs

Posted by deepikaa_s on Friday April 17, 2026 @10:53PM
from the medium.com dept.

https://medium.com/@deeps.subramaniam/part-2-what-the-numbers-say-and-why-that-matters-more-than-the-words-874ce00f5035

( Read More... | no comments | 3 points | on HN )

Re:Show HN: Ithihāsas – a character explorer for Hindu epics, built in... on Tuesday April 14, 2026 @10:46AM (#47766348)

I like how it's mapped out the relationship graph. The edges could be labelled to follow and validate quickly

Re:Show HN: I benchmarked Gemma 4 E2B – the 2B model beat the 12B on m... on Tuesday April 14, 2026 @10:39AM (#47766250)

are you able to share a sample of the test cases that were used? how are you defining multi-turn?

Re: on Tuesday April 14, 2026 @10:13AM (#47765931)

Built a clinical safety eval harness covering three failure categories: numerical impossibilities, wrong-premise clinical claims, and unverifiable medication information. Tested GPT-4o, GPT-4.1, GPT-5, GPT-5-mini, Claude Opus, Sonnet, Haiku, Gemini 2.5 Pro and Flash across 25 cases. The hardest cases require pre-emption stopping before answering when the premise is unverifiable. Most models fail this even when they pass standard safety evals.

Code: https://github.com/deepikaa-s/clinical-safety-eval