CBT is a framework more than it is a manual, and not nearly as robotic as you've described. All therapy requires a human relationship to be successful. You have to trust your therapist to believe in the things they say, and that belief is a prerequisite for putting in the effort to reshape your thoughts and attitudes, but what's so robotic about a therapist asking you to explore the origin of a preconceived notion? About them asking you to be mindful of those type of thoughts? On coaching you to intercept those negative thought patterns and replace them with improved ones?
I think chalking up the efficacy of therapy to hope and placebo is a bit disingenuous. Yes, belief in change is a precursor to real change -- no therapist can help someone who isn't willing to believe or even want to change -- and in a way, the therapist's role is to guide you in your own endogenous healing process. However, there needs to be some rigor in the way they guide you to break bad patterns and replace them with better ones.
In a way, the therapist's challenge is like inception: how can I plant something in someone else's head but make them feel like they came up with it themselves? I think most of us could look at someone with low self esteem and negative thoughts and tell them what's wrong and what they should say or do or believe instead. However, that process needs rigor, and one of the best tools in the toolbox is to increase someone's mindfulness, and one of the best ways to do that is CBT.
I think overall you're a bit dismissive of it as a shallow, robotic approach when really it's a simple, repeatable, controllable frontline intervention. It's not going to work for deep-seated issues, but I think of it as almost being a prerequisite before those deeper issues can be tackled. I am of course speaking from personal experience, in both myself and what I've observed with others.
I don't recall the name of the study, but it's described in detail in "The Heart and Soul of Change".
OCB, schizophrenia, PTSS disorders prove you wrong. And your comment about responsibility is ill-advised and bad attitude ; make a favour to people suffering from mental illness: don't try to comfort or coach them.
All therapy is "black magic" in the sense that no-one quite understands exactly how it works. CBT gets props because it's one of the few therapies that has a manual and can actually be studied scientifically.
I'm sorry but I prefer to speak with a therapist who doesn't have to consult a manual for every response he/she makes. Therapy is ultimately a human relationship, and just as messy and unpredictable as any other relationship - and subject to all the contingencies thereofe.
Just because something cannot be studied scientifically does not mean it is not effective. Evidence for the limited long term effectiveness (high relapse rates, ect) of shallow and manualized interventions like CBT is growing by the day. Compare that with more messy therapies that dig deeper (like psychodynamic) whose outcomes actually improve the longer the subjects are tracked.
The vast majority of the effectiveness of therapy is due to hope (placebo effect) and the relationship. Focusing on the "technique" is distracting and leads to a view of therapy as something that is done to you, as opposed to something that is done with you. Lasting change comes from taking personal responsibility, not from having special "scientific" techniques done to you.
That para-professionals with 2 weeks of training have outcomes as good or sometimes better than those with Ph.d's in clinical psychology is all one needs to know when deciding whether or not to categorize psychotherapy as a science.