You may not have loyalty, but if you're enjoying the fruits of society there is also responsibility to that society, otherwise one should probably extricate himself from society and see how long he makes it in the woods without it.

When I was referring to our own desires I was including human society.

The problem with psychological egoism is that it doesn't say anything.