For me it worked pretty fine careerwise (I'm now 35 to be precise). I'm working now in the technology development department of a multi-discipline CAD vendor, everyone else is really talented and I mostly enjoy my tasks. I even get to write simple linear algebra like a QR decomposition now an then. The job is stable, mentally stimulating and my co-workers are sensible human beings - who are mostly way past their 30s and some over 50s.

I turned to software engineering because computer graphics gave and give me such a huge kick. I don't know what intrinsically motivates you so I cannot comment whether I would suggest the career for you or not. Here's the rub: software engineering can be viewed in some circles as blue-collar work, which means the work environments can vary quite a lot. Also, the discipline is really young, and it shows in the amount of cargo cult and bullshit quotes one finds peddled as "best practices". There can be gigs where co-workers are experts and those can be great, however, there can be lot of environments where co-workers are perpetual expert beginners and those can suck big-time (or so I've heard).

Realistically, software engineering is still a pretty solid career choice so from a sustenance point of view I'm sure there are far worse options one can choose.