I did this once in Charlotte (which btw is normally an excellent airport). I had to wait about 15 minutes for 'the agent' to show up that would conduct the pat-down. There were about 5 TSA agents standing around doing nothing while I was waiting. Setting aside the wait time, they were all friendly enough, but they all had this look of "ugh... what is this guy's problem", and I wouldn't bet against the wait time being a purposeful deterrent.

So, while I will continue to opt out when possible, I wouldn't recommend doing it if you're short on time.