+1 for slicehost. Simple, easy VPS, just works, plenty of very handy tools. Upside of any VPS, you get to manage the whole server. Downside, you have to actually manage your server.

I've used DreamHost, but their crap up-time and extra crap support drove me out. I still use them for domain registration, but that's about the same anywhere, and transferring registrars is a little bit of a pain. If you just want an easy plug-and-play test server or a personal blog, they're cheap and not completely terrible.