I think part of the difficulty is that we don't have a great understanding of consciousness or how the mind actually works. It feels a bit like medicine prior to germ theory.

Even for relatively more understood things like depression it's not like we can test the serotonin levels to validate it before SSRIs and then check them after.

It seems to be a mixture of this seems to work, but given the high variation among people and general complexity of mental states the science is difficult.