In my experience, teaching has a way of ballooning to fill whatever time you have. Teaching one class feels like just as much work as teaching three, partly because either way you have to be available to students outside of class - office hours, answering email, and so on. I can't imagine teaching a single university class "for fun" while holding down a full-time job. Certainly it's not something I've seen very often.