There simply isn’t sufficient recorded information known about the human body for LLMs to one shot new medicines. These models are trained on known measurable information, so much of which we don’t actually know about the human body

You can’t just overcome that uncertainty with really good guesses based on what to do know. Not at least until you know a lot more.

Secondly RCTs exist because you want to capture a random portion of the population, some of who may respond differently to different treatments. That and the body is a dynamic organic system and not everyone’s is in the same state.

LLM will improve the pace at which new medicines are developed but it be a long time before reasoning entirely replace the need for empirics.