I also get what I would describe as "near" panic attacks when I smoke (about once every two weeks, with friends). I realised that after about 15 minutes or so it cools down and I feel, perhaps, more relaxed than before I started. Purely anecdotal but I feel you. Maybe a bit of cooldown and good company helps with the paranoia.