And with the example of the athlete a portion of their job is to have down time, and not be out doing strenuous physical activities the night before the big game.

When I encountered the situation where my leisure became my job, I simply didn't have anything I wanted to do to fill the free time anymore. Sure I could of done push ups or gone on dates or something else which I personally hate doing, but making myself miserable in my free time isn't my idea of healthy.