Just like you, I enjoy drinking coffee too and I don't notice any change in energy levels. If I don't drink coffee in the morning the only thing that happens is the headache in the afternoon, as you mentioned.

When I drink coffee at night, which is rare, I also don't have any trouble falling asleep.