It's the opposite with me. I can have coffee in the late afternoon, in some cases even in the evening, without problems--but if I have a tea after 3 pm or so, I'm sure to have trouble.

Might have something to do with the strength of the tea, though.