The file it saves your password in is only readable by you.

Pidgin needs to supply the plaintext password to the server to authenticate.