That is basically where I am at with my relationship to caffeine/coffee. Sleep much better without it. But life feels a little duller without it as well. And then my first cup back after a long period of abstinence is absolutely magical. That feeling quickly fades after about a week of use though.
I tried limiting it to just once a week to see if I could capture the magic, but my discipline was lacking a bit, and even when I was able to stay on that schedule, it still felt like my tolerance was building.
As an aside, I highly recommend the Audible book "Caffeine" by Michael Pollan if you haven't already heard it. It's quite brilliant.