In general, carrying things around with you are more likely to get lost or stolen in some manner than if they're just left in your room. There are probably exceptions but if I'm somewhere like Western Europe, I'm not carrying my passport around with me.