Then they aren't "chatter and hub-bub of relaxed tribemates", rather to you it is "the whispers and noises of your enemies". Of course that wont get you relaxed. The effect comes from hearing people you are comfortable around talking, if you have a distrustful personality then likely you aren't comfortable around many people but the effect is still there, just that you lack another link.