This. Bothered to login to say what you said.

We talk about AI optimized to create paper clips accidentally running amok and turning the world into paper clips. Why isn't profit a similar problem? Government agencies are just AI running on meat. Give them success parameters different than raw profits.

Honestly, why can't we see this as just a variant of the instrumental convergence problem?

OTOH, academia is supposed to be this type of meritocratic, results-driven ideal, and look where we are there. I suppose this a consequence of the measure that we use to "judge results". Do we start assigning healthcare R&D companies an "impact factor"? I know the journal impact factor is a bad example since it is criticized for being gameable, but I'm just using it to convey what I mean.

↑ ... instrumental convergence problem sounds sexy! drool