Yes, I'd only do it if I trusted the site, but it works far and away better at preventing spam than a CAPTCHA. It seems more annoying to the bad guys than the good guys. To this day it's the only thing that Craigslist has done that has worked. Phone numbers are expensive to get, especially since they block most VoIP stuff.

Google around about Craigslist phone verification and you'll read all the spammers having freak out sessions. It's a great way to spend an hour.

Definitely more annoying. I dare you to add "Phone Number" as a required field to your signup (don't even bother verifying it) and measure the change in signup abandonment.