It might be possible that it can't succeed, but we don't know that and the evidence in terms of what happens if it fails is pretty compelling, so I think morality dictates we have to at least try, at least until we have more information or another path emerges.
The first example that comes to mind is OpenAI's Sebastian Bubeck (& others) giving underwhelming examples to illustrate that GPT has surpassed human mathematicians. Or earlier, when SamA says that he has seen things you wouldn't believe, whereas the top replicant in a proper sci-fi will at least be specific about the I-Beams.
Another parallel which you would be familiar with is nuclear power. Humans can plainly see it's an unworldly tech, but I'd say the value extracted so far has been a net negative-- mainly because our collective engineering chops are just too.. profane. And SamA/Musk/Thiel/Luckey just don't seem as wise as Rickover (who is the Rickover of our age? I think the Rickover pipeline is broken, tbh)
From my vantage point, I agree with you: China sees AI as less important as solar, so charitably, I'd say that Thiel's contrarian strategy is actually to get the Chinese so riled up they zuzwang themselves into providing the progress but keeping the whirlwind for themselves (so proving he has learnt the post-mimetic game from the atomics opening)
There could be another interesting post-mimetic line exploring how a hybrid chabuduo-"festina-lente" culture is required for developing supertech.. which.. only a Rickover can play
(I don't know if you're army or navy but there's this millenniums old cultural divide between a conservative army engineering culture -- that develops atomics -- and a progressive naval engineering culture -- that makes headway harnessing it. AI bulls smell like they are more aligned with land-forces
What's doubly duplicitous is that even if LLMs achieve general intelligence, the whole idea is to enrich a tiny fraction of humanity that happens to be shareholders.
EDIT: trying to address the Roko part..I'm assuming once AGI is achieved.. the AGI doesn't need more compute to increase its intelligence beyond those of an average activist employee (I can assure you that in OpenAI there are such employees, and they know to shut up for now)
the antisocial part: it's already happening. What can you do about that.
As a thought experiment, say you were the CEO/board member of a company that's told your new platform is choosing public benefit over profits. What would you do? Now filter down the decisions going down the heirarchy, considering job security and a general preference for earning bonuses.
For all the discussions around "alignment" - any AI that's not aligned with increased profits will be unplugged posthaste, all other considerations are secondary.
What does that look like?
But LLM is not an AI. Tesla self-driving tech is not an AI.
Even my dishwasher has AI according to the manufacturer!
Can you call it a success if washing machine has AI already?
> For you AI skeptics, this is going to be rough to hear, but given the suffering that will occur if AI fails, I believe we have a moral imperative to make sure it succeeds.
What if, and hear me out here, the prediction is that it will fail because it can't succeed. It hasn't proven it is capable of producing the results that would justify itself. That there is some value doesn't mean that it will develop that capability. We have nothing to say that it will.