For me, my brain kind of locks in to the key signature eventually. It might be worth practicing scales and simple chord progressions in each key to try to imprint it in your brain. I found keys with lots of sharps and flats intimidating for a long time, too.