I actually have a very different feeling when I watch users struggle with my app. I feel slightly ashamed and very grateful. Most of the time I realise I've made too many assumptions about the UI, and i start getting ideas on how to maybe fix the problem so that other users don't encounter it.