To be fair, my experience with them is mostly playing with them in the store, but as you say, that's where it counts. In day-to-day usage it tends to be less of a big deal. You adjust how you use the device.
I'd argue the opposite when it comes to UI. If you're tapping on the same link 2-3 times bc you're not sure whether Android/iOS registered the tap, that's an intractable problem that gets worse with time. You can't adapt your behavior to that annoyance.
iPhone users, for example, don't seem to mind life without a back button. It becomes a fact of life and users adapt their usage of the device accordingly.
On the other hand, when I updated my iPhone 3G to iOS 4, the responsiveness of the UI suffered by orders of magnitude. After more than a month of usage, I still found things like typing a text message excruciating. (Mostly fixed with the 4.1 update, praise god.)