Having done none of these, perhaps it's something about the quality of the experience? E.g. perhaps when you do cocaine/heroin, the quality of the high is so clearly unusual and extreme that you could never mistake it for being an accurate representation of "how things could actually be" without depression/anxiety? And maybe shrooms etc aren't nearly as extreme?