I'm wondering whether there are any Android games that'd use custom formats or whether lots use off the shelf engines.

I've always had a passing interest in game reverse engineering - I remember the days when "ripper" applications had a good chance of pulling the music out of a game.