Just another human still imply's you don't relax by selling yourself on fire. Specific instincts are not quite universal, but they very much shape behavior.

Oh ok - if your point is simply that an AI will probably have some set of goals which could be called values, we agree. I was attacking the common claim that superintelligence will inherently develop 'friendly' values.

The orthogonality thesis doesn't say AI will lack goals, only that those goals may be totally unrelated to the sort of beliefs you or I would recognize as morals.

Actually, I am saying that there are some universal morals which AI will also come to understand. They may not be your morals, but I think they are compatible with mine.

The morals we have work; that's why humans developed them. They seem to essentially amount to giving a little leeway to allow cooperation, even if it allows yourself to be taken advantage of, avoiding a constant race to win and giving up some personal gain for the benefit of the group.

It's mostly long-term game theory - and AI will be subject to the same laws of math, physics and group dynamics. If they don't try to get along, they'll reach an equilibrium of constant war. Those groups of AI who develop morals will avoid that and surpass the other groups.