I've gone on anti-sugar diets too. This issue I'm having is definitely from the caffeine.
I get those symptoms from coffee. I don't eat processed sugar and I mostly avoid other carbs during the work week. When I don't drink coffee I don't need an afternoon nap. But I like both coffee and naps so I often have coffee.
Do you put sugar in your coffee, though?
What you're describing is sugar dependence, not caffiene. Cut out all processed sugar and after 2 weeks you'll wonder what drug you are on (then cut out "healthy" carbs like bread and pasta and you'll wonder how you ever wanted to eat them).