There's a greater chance in most places to have your passport lost or stolen from you than losing it from somebody breaking into your hotel room I think (zero data to back this). It's also a hassle; you want to go to the beach and swim or snorkel or any sport, now what do you do, having to keep an eye on a pouch or something.