Having done both far more than that, that's not my experience. I have had wildly visual mushroom trips (at higher dosages) and subtly visual LSD trips (at lower dosages). At comparable dosages and settings the main difference to me is just the duration.

Mushrooms in particular are hard to standardize a dosage of. Eat enough of some strong ones and things get plenty visual.