I have a friend who works in finance at a huge bank. He makes a killing, but absolutely despises his job. His stated goal is to make enough to retire early (we're ~40 now, so soon-ish) and then work part time at a hardware store just for something to do. Pretty much for the reasons you said, meet people, low responsibility and stress, the job ends when you punch out.

I think I'd get bored pretty quick, but I do understand where you're coming from.