As your last statement here illustrates, this is because of the way most health care is paid for. In a system where third parties pay for most health care, the doctor has to answer to the people who pay even though they are not the patient. Even if he personally is convinced that a particular experimental treatment is low-risk and has some chance of success, he still has to convince the people who pay, which is a much higher bar.