Maybe your consciousness does sometimes occupy other bodies. Of course you don't remember it, because memory and working memory are tied to brains. When consciousness is in another body, it feels that it is another person with his memories and context and none of yours.

Another way of saying this is that there's no need to postulate separate consciousness for separate people, one is enough.