I'm not as jaded, I've been lucky enough to work at nicer places, but having made the same jump I would strongly recommend the same thing.

Either nights and weekends, or after a few years as a SWE you could just retire somewhere cheap and go nuts on your game dev passion.