No, caffeine is physically addictive. Your brain starts to develop more adenosine receptors (since you keep plugging existing ones with caffeine). When you stop drinking coffee (or the side effects get to be too much to keep up, etc.), you get hit by an avalanche of adenosine, and, suddenly you're without protection. Physical withdrawl symptoms ensue.

It's not just a behavioral thing. The brain physically adapts to maintain equilibrium in the recurring presence of caffeine. I don't think there are permanent changes, but the adjustment cycle lasts days. The half-life of caffeine is 6-8 hours (on average), meaning, at that point, half of it has been metabolized.

Anecdotally, a reasonably small amount daily is stable. Something like two cups of coffee (one in the morning, one after lunch) is fine for most people. Different drinks have different amounts of caffeine, of course. (Also, tea seems less likely to make me jittery than coffee.)