One more factor, besides the ones other posters have pointed out, is tolerance. I find my tolerance to the anxiety provoking effects increases faster than my tolerance to the anxiolytic effects. If I haven't consumed any in a while, I find I can get that initial panic. Ride it out though and the more relaxing effects take over.