The biggest factor for me is the games, though, and only a Nintendo platform will ever have Nintendo games (industry trolls that keep calling for Nintendo to become a third-party developer notwithstanding). Nintendo is definitely one of the top game development houses out there, and Apple doesn't really have anything that can compete. Nintendo understands games, Apple does not.
I don't think the iPhone could ever dethrone the PSP or NDS, but I think if one already owned an iPhone that they'd probably ignore the other gaming devices unless there was a specific game one wanted to play. I fall into that category (as well as the significant other who likes to steal my phone), and I definitely haven't looked twice at any gaming specific devices because I can get everything I want on my phone most of the time. I only pack the NDS on trips involving kids I don't trust with my iPhone :)
I think the DS would be more likely to be successful adding calling features (like the PSP has with full-on Skype support) than an iPhone would be adding real tactile gaming controls and becoming a legitimate platform.
I get it - I get it - people want the iPhone to be the magical "one device". But I'll tell you what - Nintendo has never lost a handheld generation (Lynx, Game Gear, and PSP to date) and they don't look like they're going to any time soon.