I was (and am) fine with no physical keyboard, but omitting a GPS chip was just plain goofy, considering how nice the Maps application was. Others were complaining about the first iPhone's lack of cut and paste functionality relative to some other phones, the inability to replace the battery or add external storage, the inability to use third-party applications, and the substandard camera.
Bug-wise, if you could get Safari to avoid crashing back to the desktop for more than 30 seconds at a time, you were way ahead of me.
All of this stuff got better in the next generation. It was Apple's dedication to fixing the phone's shortcomings that really made me a dedicated iPhone user.
I also bought a 1st gen iPod touch on the week it was released and never had any serious issues with Safari other than browser compatibility.