It's very very unclear. One person can hit heroic doses (200-1000 micrograms of LSD or equivalent of mushrooms) and be completely fine afterwards, another can have a small, usually safe dose and have long-term issues. There are studies done to detect who will be susceptible to this, but there is no clear answer yet (aside from people with family history of bipolar or schizophrenia).

More likely too frequent use. Psychedelics twist reality, doing them too often definitely leaves a mark. Eventually you have trouble telling what's real and what's not even when you're sober.