Recently, I started creating a throwaway physics/game engine for fun and it is an awesome way to learn some really cool algorithms. Also, I did a small image manipulation app that exposed me to some cool stuff as well.
As a new grad it definitely pushed me more towards game development- would you mind sharing your reasons for leaving?
It's also tough for some engineers because there's usually a very clear product vs engineering divide -- you're implementing stuff other people come up with, and when they change their minds you throw it out and do it again a different way.
That said, making games is also a ton of fun and is a great way to be exposed to a lot of interesting engineering challenges. If you like games and you can put up with the issues above, you should definitely consider it.
I spend less time these days implementing algorithms. However, you need a good understanding of algorithms to decide between using ActiveRecord or a service layer. You need a good understanding of algorithms to pick between a relational DB, a key-value store, or a document store.
Maybe you aren't picking difficult enough problems if you don't have to think about this kind of stuff.