The heart rate thing sounds like it is what gets me, like a moment of panic of, “oh shit I’m about to die!!” The times where I have pushed past that and chilled out it was more enjoyable, but even knowing it’s not going to kill me, I can’t help but worry I’m the first person who is going to OD on a few puffs of a joint.
Fear and panic in combination with an unshackled but inexperienced mind has the potential to create your own personal hell in no time.
That's also a good reason to get introduced to this class of drugs by responsible, experienced users who can help guide you out of any loops you get stuck in.
But it's likewise very common to use Cannabis to take the edge of the unsettling feeling of having your ego deconstructed/reconstructed when going in/coming out of stronger psychedelic trips.
Also worth noting that more Sativa-leaning hybrids are potent psychedelics in themselves at higher doses.
The other source of anxiety is that cannabis raises your pulse for a few moments, I would guess because of increased blood flow (which is sometimes visible in the eyes). From my experience that never goes away entirely, but you learn to not freak out since it's not dangerous.
Organic drugs like shrooms are definitely less dramatic than synthetic such as LSD from my experience.