Holy shit. I've always been allergic to random things like cherries, apples, tomatoes, melon, peaches, etc., but only some of the time and never when it's cooked. It could definitely be some other tree or something that's doing it.

Edit: Just read the other reply. I'm definitely allergic to birch and grass pollen and I've always had hay fever. That solves that mystery.