To help you conceptualize this as not a problem, I'd say you'd be in the same boat with anything.
Many hobbies take time to prep, do, then clean up after. EG, working on a car/boat, working with hands, or really many hobbies.
Jumping in/out isn't doable. And yes, I know this is a sadness.
But a full life is its own reward too...
(To me Elite Dangerous is the VR experience. Anyone who love space and have a chance to try E:D in VR absolutely should do it. I refuse to play E:D in 2D, and I'll probably give Frontier money for the new expansion even if I will not play it for many months just in hope that the game will stay around in years to come to have a chance to scratch my space itch)