While the lack of tactile controls is a big downside to the iPhone as a gaming device, there are plenty of games that are playable without, and for very casual gaming for short periods of time even the ones faking controls on the touchscreen can pull it off.

I don't think the iPhone could ever dethrone the PSP or NDS, but I think if one already owned an iPhone that they'd probably ignore the other gaming devices unless there was a specific game one wanted to play. I fall into that category (as well as the significant other who likes to steal my phone), and I definitely haven't looked twice at any gaming specific devices because I can get everything I want on my phone most of the time. I only pack the NDS on trips involving kids I don't trust with my iPhone :)