>- "Paperclip maximizer". A corporation is a paperclip maximizer. If you take Milton Friedman seriously, that's what a corporation is supposed to do. As I point out occasionally, most of the ethical problems foreseen for AIs are known ethical problems with corporations.

Corporations are not capable of having the "fast takeoff" which is an integral part of the doomer narrative here. Furthermore, a corporation being a paperclip maximizer is an abstraction that obviously falls apart in the extremes, if for no other reason than because at some point destroying the world for the sake of more paperclips is going to have a negative ROI....

Meanwhile, the (AI) paperclip maximizer is actually trying to maximize paperclips and does not have the sort of internal checks that would apply to real-world orgs. That is the crux of the problem.

Corporations, when first developed, took out large chunks of Asia before anyone could stop them. The main difference is that corporations have to be powered by people while an AI could theoretically work with complete independence.

I think the argument is that due to emergent behavior and possibly perverse incentives corporations are indistinguishable from slow-moving AI -- they can be treated like independent agents.

Elsewhere in the thread, it's brought up that AI could move faster than corporations generally move. That might be true, but there are still physical limits: the speed of light, moving mountains of steel takes energy, etc.

> Corporations, when first developed, took out large chunks of Asia before anyone could stop them.

What do you mean by this?

British East India Company, probably?

At some point it had more soldiers and controlled a greater subject population than the government of Britain itself.

I'm not sure it was ever profitable, though, which is the main paperclip corporations ostensibly maximize. Which itself speaks to the problem: superficial objective functions don't always determine outcomes or behavior.

Beic was only a corporation in the most technical sense of the word. It became so powerful because the Queen had given it a government enforced monopoly from the start and the government kept extending that monopoly in return for loans and shares of the profits. The best modern equivalent would be Saudi Aramco, which also has its own "army", but it's meaningless to compare these sorts of organisations to modern uses of the word consideration. They're much closer to government departments.

What exactly is the argument that corporations are incapable of unbounded exponential growth contra a possible future AI? Is there just something magic about computers, or am I missing something obvious?

> What exactly is the argument that corporations are incapable of unbounded exponential growth

Individual corporations aren't capable of unbounded exponential growth because they can't keep the interests of the humans that make them up aligned with the "interests" of the corporation indefinitely. They develop cancer of the middle management and either die or settle into a comfortable steady-state monopoly.

Market systems as a whole can and do grow exponentially - and this makes them extremely dangerous. But they're not intelligent and so can't effectively resist when a world power decides to shorten the leash, as occasionally happens.

You can find discussions on this by googling "AI Foom" - the key idea is that an AGI that can recursively self-improve will be able to rapidly escape human control before any entity could check it, and very likely without humans even knowing that this had happened. Such an AI would likely consider humans an obstacle to its objectives, and would develop some sort of capacity to quickly destroy the world. Popular hypothesized mechanisms for doing so involve creating nanomachines or highly-lethal viruses.

That's key to the story of the paperclip maximizer - the paperclip maximizer will go about its task by trying to improve itself to be able to best solve the problem, and once it's improved enough it will decide that paperclips would be maximized by destroying humanity and would come up with a plan to achieve this outcome. However, humans may not realize that the AI is planning this until it's too late.

Corporations innovate through the work of human minds. Corporations improving doesn't cause the human minds to improve, so there's no recursive self-improvement. Corporations today still have the same kind of human brains trying to innovate them as they did in the past.

A human+ level AI would be able to understand and improve its own hardware and software in a way we can't with our own brains. As it improves itself, it will get compounding benefits from its own improvements.