One could argue that this is the problem. Not that you spend your time doing activities others wouldn't consider leisurely, but that you feel compelled to because otherwise you become "really anxious." Some anxiety can be a propulsion, but too much can be a pathology. I'm certainly not fit to evaluate your own, but by your own words I'd suggest it's possibly worth further reflection.
Check out Four Thousand Weeks for some recent musings on why that anxiety may be best ignored rather than heeded: https://www.goodreads.com/book/show/54785515-four-thousand-w...
There's the "simple carbohydrate" fun of, say, an action game, which is immediate, easy, but low-nutrition and, enjoyed overlong, makes me feel empty inside.
Then there's the "complex carbohydrate" fun of learning something new or working on a project, which is more fulfilling and interesting but also work -- mentally taxing -- and, enjoyed overlong, burns me out.
My brain seems to demand a balanced diet of both.
What happens when you get to such a state where you simply cannot NOT waste time? E.g. your job becomes so soul-sucking that you have no mental energy left? Might you be then the author of the post? Hmm.