So instead of a super intelligence either killing us all or saving us from ourselves, we’ll just have one that can be controlled to extract more wealth from us.

IMO, this is the most likely outcome

I largely agree, although I do see how AI can help with social coordination problems, for example by helping elected leaders be more responsive to what their constituents need. (I spend a lot of my own time working with researchers at that intersection.) But social coordination benefits from energy research, too, and from biology research, and from the humanities, and from the arts. Computer science can't singlehandedly "solve" these problems any more than the other fields can; they are needed together, hence my gripe about total-orderings.

Exactly. Or who gets the results of its outputs. How do we prioritize limited compute?

Even not just the compute but energy use at all. All the energy burned on training just to ask it the stupidest questions, by the numbers at least. All that energy that could have been used to power towns, schools, and hospitals the world over that lack sufficient power even in this modern age. Sure there's costs to bringing power to someplace, its not handwavy but a hard problem, but still, it is pretty perverse where our priorities lie in terms of distributing the earths resources to the earths humans.

Unused electricity in one location is not fungible to be available elsewhere.

No, but the money used to build the power plant at one location was theoretically fungible.

Norway choosing not to build a plant will not result in Guatemala getting that money instead.

Social coordination problems exist within a specific set of constraints, and that set of constraints can itself be altered. For instance, climate change is often treated as a social coordination problem, but if you could produce enough energy cheaply enough, you could solve the greenhouse gas problem unilaterally.

OK, lets play this out.

Lets say an AI discovers cold fusion. Given the fact that it would threaten to render extinct one of the largest global economic sectors (oil/gas), how long do you think it would take for it to actually see the light of day? We can't even wean ourselves off coal.

If you had enough energy because of “cold fusion”, you wouldn’t need the world to switch to it. You could just use it to synthesize hydrocarbon fuels from the air and water. I don’t think this is a panacea, but in principle, if a single medium-sized developed country actually had a big enough qualitative breakthrough in energy production, they could probably afford to either undercut the actual oil and gas industry or else just eat the cost of sucking all the extra CO2 out of the atmosphere. (Personally I think there are other possible approaches that may present easier solutions or mitigations; I don’t think this is literally the exact way we end up addressing climate change.)

Part of the supposed “social coordination problem”, from a certain perspective advanced by some environmentalists is the belief that ordinary people need to make serious qualitative sacrifices in their lives for the sake of stopping climate change. They might retreat to the motte of “the oil and gas companies are running the world” but in reality, their version of the “social coordination problem” is that ordinary people like being able to drive places or cook on a gas stove, and they’re frustrated and annoyed that they can’t scold them into abandoning these preferences. I personally am of the opinion that these people are creating more social coordination problems than they’re solving.

Nonetheless, as far fetched as it might seem that technological innovation can solve these problems, it seems even more far fetched that social coordination is going to solve them. And there’s a long history of these types of problems being either solved or made completely irrelevant by technological change. Which sometimes introduces completely new sets of problems, but then those get solved too.

Play that out. How do you see that going? Some researcher post doc with access to ChatGPT-12 just casually asks "hey this thing that violates the laws of physics as we understand them, how could that work?", and ChatGPT-12 says "oh that's easy, just frobb the frozzonator", and the post doc just wanders over to Home Depot, grabs the parts to build a cold fusion plant to do that for $50 on a post doc's salary, but is killed by the evil agents of Big Coal on the drive back to the lab? How would that work?

Every idiot who doesn't understand the science and math behind fusion can now get ChatGPT-4o to give themselves an interactive review on the physics and practical reasons why cold fusion doesn't produce more power than you put in. (Muon-catalyzed fusion is a room temperature science thing we can do, but it won't produce power so it's relatively uninteresting).

If cold fusion were possible and a postdoc figured out how with the help of ChatGPT-12, they'd announce it on Threads, write a draft paper, run the theoretical physics past everyone will listen, everyone wound rush to replicate and confirm, or disprove the theory, funding would roll in when it was actually agreed to be theoretically possible, we'd build it, we'd have limitless power too cheap to meter, and then we'd argue over if ChatGPT-12 deserves the Nobel or the postdoc.

Cold fusion isn't being held back by Big Coal.

there's a huge gap between discovering something and it becoming widely available

just look at the slow adoption of EVs in the US due to the lack of infrastructure. Eventually it might gain parity with ICEs but we're a lot further away from that than people anticipated.

> I am skeptical because the vast majority are social coordination problems, and I don’t see how

Leadership.

By any means necessary I presume. If Russian propaganda helped get Trump elected, AI propaganda could help social coordination by influencing public perception of issues and microtargeting down to the individual level to get people on board.

could but it's owners might have a vested interest in influencing public perceptions to PREVENT positive social outcomes and favor the owners financial interests.

(seems rather more likely, given who will/would own such a machine)

are humans smarter than apes, and do humans do a better job at solving social coordination problems?