What of drinking alcohol? Eating too much food? Riding a bike without a helmet? Taking an antidepressant with known potentially dangerous side effects? Going outside without a jumpsuit made out of Nerf foam? At what point do you get to live your life for yourself instead of avoiding doing things you think others might not want you to do?
I think jumpsuits should be forbidden for the same reason, but that's just a personal opinion.
Given the chances you'll be riding your bike among cars piloted by people on prescription painkillers or alcohol, I'd be surprised if riding a bike isn't significantly more dangerous than taking psilocybin, but once again, there are no good stats on psilocybin-related behavioral results so this is just speculation.
Edit: the OP removed statement about addiction