Not impossible: you could get the application specific password and then start using it on other sites.

That would be foolish, but users do all sorts of foolish things.