Same, as well as many of my friends. I started off by trying to make simple games, then playing with game engines, then playing with the internals of game engines (physics engines are so freaking cool!), which lead to just playing with those pieces of code/math, sans any game.

Basically "I want to make a badass action game" eventually turned into "I want to make a badass graph-traversal algorithm"