iPod Touch (yes, Apple still sells them, starting at $199)? You’ll get a camera that’s worse than in a phone, though (8 megapixel) and ability to run lots of apps. I think you can remove or hide most of them, and configure it to disallow lots of things, though.

The lower quality camera likely will be the deal breaker.