I think finance is a great place to work (yes, even as a full-time programmer) if you have a strategic reason for being there (VCs like financial experience, there are some interesting machine learning problems) but I would say that, in general, you're best off if you know you have the social and political skills to get the best work and, if you don't, finance is not likely to treat you well.
I worked exclusively on front-office systems nearly for the entirety of my time as a developer for an investment bank. I didn't even get into how there are very few opportunities for growth and other, more HR-related/culture aspects of the job.
I'm glad you have good things to say about your work! I just wish I could describe the time that I and my colleagues spent at those banks in a better light. It was utterly life-altering when I left. I felt like I was living for the first time since college.
Then again, the culture and the work varies in different cities and in different roles. In London we tend to get excellent developers joining the industry (traditionally there hasn't been much else for good developers locally) so you can typically expect to have skilled colleagues. And the closer you get to the front office, the more money there is for IT projects, and the more remit you have for writing good software.