Hmm, I'm not sure I understand this line of thinking. Intelligence is complex, soft, and can even conflict with itself. Why do you envision artificial intelligence as if it were a pure optimization algorithm with high-level problem solving abilities? It seems to me that a fuzzy metaheuristic with lots of competing goals would be a better comparison and this does not lend itself to the "paperclip factory" quite as readily.
The irony about these conversations is that humans are already far more of a threat to humanity now than an AI might be, possibly, at some undetermined time in the future.
AIs (that don't exist yet) are going to be in charge of controlling electronic systems (examples given). They will behave with a time-constant that is unrelated to human reaction times. They will be able to do harm, and compound that harm, at rates mere humans will be powerless to react/
1) The main approach to creating weak AI that actually works is to define some objective function and train some sufficiently complex mathematical model to maximize that objective. Hence it seems likely that if strong AI is created it will follow this paradigm and therefore such a strong AI will care about nothing other than how to maximize its objective. Such a strong AI would be nothing like a human mind so anthropomorphised thinking is of little use in reasoning about it.
2) Given a sufficiently powerful optimization algorithm (say, for example, if P == NP and an efficient algorithm for an NP hard problem were discovered) one can in fact imagine the above paradigm producing an incredibly powerful intelligence, one that could discover many sub-goals for its fundamental optimization objective which the creator of the fundamental objective could not foresee. Further it could be exceedingly effective at achieving its sub-goals, much like modern chess programs are exceedingly effective at winning at chess, but in a far more general domain.
There are however reasons to doubt such a danger.
1) If P != NP then although it should in theory be possible to achieve roughly human equivalent AI, the sort of truly superhuman AI that some fear may not in fact be possible. In other words it may be that if P != NP there is some limit to how powerful an intelligence can become due to the combinatorial explosion that occurs when one tries to expand the range of possibilities that are considered.
2) Even if no such limits exist there are probably ways to maintain control over even a superintelligent AI. A powerful optimizer would be exceedingly good at finding solutions within its own model of the world but that is only part of what is needed to construct a super AI that actually interacts with the world. You would also need a process for creating and updating the world model based on empirical data. This type of architecture would probably give you several powerful levers for controlling the AI and you could always let the optimizer run on the world model to check on its behavior before allowing it to act on the real world.
Regardless I don't know why you think a messy AI is safe. Having a really complicated utility function doesn't mean it is going to result in anything desirable for humans.
E.g. if the AI decides to use the mass of the Earth to build a dyson sphere around the sun. Or tries to build the biggest computer possible and turns the Earth into a giant computer. Or it values reproduction and tries to replicate as much as possible and outcompetes us. Or it values self preservation and kills us because we could make other AIs which could threaten it.