voidmain

You are not the farrier in this analogy. You are the horse.

Worse, P(doom|corrigible human replacement level AGI) is also very high, because the dominant strategy in international competition will involve zero humans. So we are screwed even if "alignment" turns out to be surprisingly easy or if there is a plateau right around human level.

It's butlerian jihad or bust, I'm afraid.

The most likely path I see for human survival and flourishing is for AI to do something in the near future so destructive and terrifying that it unites humanity behind permanently banning it at any cost.

It therefore seems like "AI safety" people may actually be working against our safety.

Sort of like how the human economy is limited by the fact that ants can't afford our products and services?

The first thing I do when I see an interesting article title is click on the comments to see if people have noticed that it's AI generated. If I didn't have this option because of the policy you desire, I would probably just stop reading HN entirely.

I'll defend i^4x since I like it better.

(cost x, sint x) is a point on the unit circle x turns counterclockwise from (1,0).

cost x + i sint x is a point in the complex plane x turns counterclockwise from 1.

Now look at integer powers of i, a point in the complex plane 1/4 turn from 1:

i^0 = 1 (0 turns from 1)

i^1 = i (1/4 turn from 1)

i^2 = -1 (2/4 turn from 1)

i^3 = -i (3/4 turn from 1)

and we define complex exponentiation such that, for all real x,

i^x = cost (x/4) + i sint (x/4) (x/4 turn from 1)

How about i^4x? It makes it clearer which direction we are rotating (vs -1 which is a 0.5 turn rotation in either direction) and avoids the garden path confusion that could arise from it not being obvious from the left side of the equation that we are working in the complex plane at all.

> AI writes copy like this because that's how copywriters did it in the training set

I don't think this is true. Someone probably had to write like this, but a base model has to be able to write like anyone. The specific mannerisms of AI appear in post training, either as things that are rewarded or as mode collapse artifacts. ChatGPT didn't get obsessed with goblins because that's incredibly common among humans.

The visualization is of the power set, which is uncountable.

> Q: Do you think KVM vulnerabilities will keep appearing?

> A: Yes. I recommend establishing a sustainable patching process for host hypervisors. Winter is coming.

Vibe coding is doing its part to keep linguistic diversity alive: My wife went to sign up for a doctors appointment, and the form offered choices for "preferred language" that included Old English, ancient Egyptian and Elamite.

I think you have slightly misunderstood the halting problem. There is no computer program H that, for every program P, determines whether P halts. This does not imply that for some particular program Q there is no proof that it halts. In fact in the setting where programs do not take input there is always such a proof (just a trace of the program running to completion). And there are many valid proofs of theorems like "For all x, Q(x) halts" for particular Q. Consider, for example, the case where Q has no control flow at all!

Again, my concern is about things that humans will order robots to do, for fear other humans will do unto them first, and does not depend on a loss of control. I think loss of control is not too unlikely in the process of such an arms race, but it hardly matters.

You don't need superintelligence to get this problem, but you do need human level intelligence across all domains. I don't know how far we are from this, and neither does anyone else.

Governments, initially made of humans, will deliberately create military and industrial robots and assign them to create and scale production chains that go from natural resources to finished war machines. Maybe your government will not do this; in that case it will be helpless against the robot armies of those that do (yes, even if it has a few or few thousand nukes).

Let's ignore all the unnecessary cruelty throughout all history and the present and assume that you are right. It doesn't matter, because people will be forced into these choices by competitive pressures even if their intentions are good. If your government tries to build a wonderful utopia for humans with the power of AI and robotics, it will be devoting a smaller fraction of GDP to producing robots that produce robots that produce war robots, so its robot army will be exponentially smaller than a different country that devotes a larger fraction.

The reason we have some nice things in the modern world, flawed though it is, is because for the last few centuries the military-industrial meta has favored countries which let people have some political and economic freedom. In a previous era, the meta was knights and peasants; a country without knights would be conquered, a country without peasants would starve, and a country that did not give the knights power over the peasants would lose a civil war. It doesn't matter what people want; technology will diffuse, someone will play the game to win, and any countries that choose less optimal choices will just be swept away.

Nukes are not magic wands, they are mostly only worth it for targeting cities and other very dense concentrations of value and production. An enemy state going down this path will probably start out with cities full of humans (and will probably want to "mine" those cities for materials), but its growing robot production will be more broadly distributed (and EMP shielded) and therefore much harder to destroy. And they will of course have their own nukes. I agree it's possible that a global nuclear war, if it starts early enough, ends with no one still having the ability to operate AI. But that's not exactly the most attractive way to shut it all down, is it?

I already live off my wealth, and I expect AI development to destroy my life.

Once humans can be robustly replaced by machines, the military-industrial "meta build" will be a state with no humans, no cities, and 100% of all production dedicated to war. Whichever states descend to this new equilibrium fastest will crush the others and then fight over the corpse of the world forever. (Unless, of course, something like the Yudkowsky scenario happens along the way, which is totally possible)

I can't understand why anyone expects things like civil rights, property, the rule of law, nuclear deterrence, democracy, etc to survive when human beings no longer need each other. Why do investors in AI companies expect a payoff once they are no longer needed? Why (and, more importantly, how) does anyone expect a UBI from a government to which they are nothing but a drain on resources and which is in existential competition with other governments with exponentially growing robot armies?

Either the world will come to its senses and shut it all down, or there will be no winners at all.

The safety of safe rust is created by extremely strong and subtle invariants that unsafe rust must maintain perfectly (or all bets are off). The idea that a program with tens of thousands of lines of vibe coded unsafe Rust is memory safe is mind boggling.

Nuclear weapons are mostly a threat to cities, and a zerg rush build would just not build any.

I think that if all we get is skill cloning for formally verifiable tasks, that is not "AGI" and does not doom us. It probably won't even doom software engineers, because their whole job is also not formally verifiable. I don't trust that the next ten trillion dollars of AI research won't give us data efficient continual learning and agency on par with humans, which is what I would call AGI.

There is no doubt that compressing a whole industrial supply network into a little probe is incredibly hard.

But I can't see microgravity specifically as a huge challenge. If you can get a probe to another star system, you can probably figure out how to spin it.

If AGI means that AI+robotics can robustly substitute for human labor, and robots are cheaper and faster to build than humans, then (a) anyone ruthless enough can zerg rush and defeat any nations that don't discard humans, (b) no one without a massive robot army will be needed in any way by their rulers. If this isn't a recipe for a horrific outcome, what is?

We are really getting to the point where the tech industry must be stopped if humanity is to continue at all, let alone thrive.

> In The Argument’s newest poll, fielded nationally among 3,008 registered voters between May 29 and June 3, 2026, over a quarter, 27%, of voters said it’s likely humanity would go extinct because of AI. At the same time, just 6% listed the technology as one of their two top issues — exactly one-quarter of voters think humanity could go extinct but don’t rank it as an important issue.

:-/

Providing access to the public is presumably just a temporary stopgap for these companies until they have robot armies to replace their customers and employees.

I think the tiny size of a nuclear weapon and very short interval of nuclear reaction before "disassembly" mean that even though the energy release is small compared to an asteroid impact the temperatures are probably much higher.

(I'm not an expert, though, this is a guess)

Democracy is also doomed by sufficiently capable AI. When the "meta" military unit was a knight in shining armor, most societies were under feudalism, ie rule by knights. When guns became cheap enough that whoever had the most guys would win a civil war, we got democracy: rule by whoever has the most guys. When whoever has the most robots will win a civil war, what kind of government do you expect?

Follow this reasoning to its conclusion: once humans are no longer part of the most efficient military-industrial "meta build", states that keep them alive will be outcompeted and eventually destroyed by those that do not.

But you sacrificed some of your 33 bits of anonymity to have this setting work as intended. And that isn't strictly necessary: the web could have been engineered so that the selection of light or dark styles takes effect in a way undetectable to a web site.

Making me talk to a fucking robot leaves me with a deep and abiding hatred for your company. I will prefer almost any alternative to doing business with you and hope fervently to read about your bankruptcy.