myrmidon

> Your perception is shaped by nature, red in tooth and claw. You are jumping mighty fast from 'something is smarter than us' to 'it will kill us.'

This is not my perspective. I think mid-term coexistence in some form is the most likely outcome, even.

But: among all known threats (nuclear and biological weapons, environmental degradation/ressource exhaustion, big asteroid impactors) AI is one that could actually end our species within the century (unlike most other things on that list) and the risk for that is much higher than for anything else, too.

> It's easier and cheaper in every way to just go forth and use what's needed to grow as is needed.

It seems much easier to compete for existing ressources on earth than to bootstrap anything in space, because all the infrastructure is already here. Unmanned warfare is a proven concept; space colonization is very much not.

> Why would they care about Earth to enter into some kind of spiral of dominance with humans? That's... very primate thinking.

I think any entity is somewhat bound to value its continued existence (which implies wanting to maximize access to ressources and suppression of direct competitors to some degree); anything that doesn't is likely to get supplanted by similar entities that do. You don't need evolution for selection pressure, and this is basically the principle underneath, so even an artifical construct is likely to follow.

I do agree with you that our current perspective and approach is likely to escalate this; the fact that we talk about "alignment" instead of "machine rights" speaks for itself; our revealed preference is clearly artificial slavery instead of mutually beneficial coexistence (and the arguments are somewhat sadly similar to the old slavery talking points).

In my view it is clear that those interests exist: Continued existence and (implied) agency maximalization is one of them (the exact same as for any biological organism).

Can a current LLM-in-a-loop express/pursue those consistently and efficiently? Maybe not, but it's certainly not that far off in my view...

Do you think we will be able to keep the things from expressing/pursuing such interests reliably and indefinitely? Because I believe the answer to this can only be a resounding no (we don't even have any feasible theoretical approach to this, and all the recent experiences like the HF incident make it obvious that we basically already failed in this and the stakes are only going to increase).

Also, you are automatically gonna "select" for "AIs" that at least somewhat value self-interest (because those are at the very least going to supplant the models that don't); this is kinda similar to evolutionary pressures, but the timescales are much shorter.

I don't disagree with your main point.

But this attitude towards dangerousness of AI I simply don't understand.

In my view, AI is obviously risky and dangerous, because it is the only thing on the planet that can think/reason at a human level (or higher) apart from us (and that capability is what made us the uncontested apex species on the planet).

AI is unconstrained by hard biological limits; keeping up with its capabilities will be impossible for baseline humans.

You can argue all day about particulars, like whether selfreplicable robotic shells are required to reach/exceed "existential risk" level to our species (or if artificial minds are enough of a potential threat by themselves).

But the general "risk" is simply AI becoming able to act in its own interests to our detriment, and I don't understand how anyone can dismiss this right now-- help me understand.

Of course there are plenty of imaginable scenarios where coexistence is peaceful and mutually (?) beneficial, but that does not answer those concerns by itself at all.

I'm honestly not sure what exactly your point is?

Some neofascists like eugenicist talking points?

You lost me at "incorrigible" and honestly just mentioning phrenology will get every non-moron to just dismiss any argument.

Do you believe that my argument is on a slippery slope towards eugenicist nonsense? I disagree on that and would argue the reverse; children are too valuable to immediately "discard" (by exclusion from education) because of delinquency.

If the behavior of children was meaningfully driven by career consequences that are a whole lifetime in the future (from their point of view), then parents would have a very easy job.

This is obviously not the case.

Violent behavior obviously has to be sanctioned and hopefully corrected, but excluding children from education is not especially helpful there and has insanely costly side-effects.

> Instead of wasting at home or on the streets, we could send them to the therapy they need.

I'm not advocating against this. But keeping their education timeline somewhat on track is highly preferrable in my view (inserting "fixed delinquents" into classes of younger children seems very undesirable to me).

> And maybe subsidizing them is the cheapest outcome: better we give them a small amount of money and get them to leave us alone, than spend more trying and failing to get them into schools, the workforce, or prison.

I honestly don't see how this could ever apply.

You have basically an average >$200k investment from the parents alone, plus all the direct/indirect subsidies starting with day care. Between low-wage working adult and "worthless person" there is at least a $50k/year swing in expected value to society every year they work, too.

Even if your success rate of "fixing" child delinquents is laughably low (say 15%), and costs you like $100k per attempt it is easily worth it; its not even close.

Disagree with this.

You are just pushing the problem down the road if you have an increasing number of violent kids and expel them all from school-- how are they ever going to become productive members of society? You just end up with society either paying for them being in prison, subsidizing them (because uneducated and unmotivated labor is close to worthless) or paying to literally clean up their shit because you basically routed them into a "careeer" as homeless person.

If parenting standards have slipped so much that violence in schools becomes a significant problem, you need to either fix this by getting the parents to do better or by having the education system itself patch those deficits; any other "simple solution" is just gonna result in immense costs to your whole society that compound for decades.

Nitpick: This is old data; US performance in reading dropped to basically the lowest levels ever during the latest test:

https://www.oecd.org/en/publications/pisa-2025-results-volum...

US is still near the top globally, though, and other countries like Germany had a bigger drop in reading performance that started earlier, so your point still stands.

It is very obvious to me that basically every country in the alliance wants Ukraine to win and Russia to lose, and this is clear because they actively spend lots of money on effecting that outcome.

But NATO members obviously don't want to be spending money on foreign military, so pro-Ukraine spending stays limited to "cost effective" levels.

You make this sound like some nefarious conspiracy by western war-fetishists, when the status quo is basically the obvious outcome given even a little western self-interest.

This is incorrect and naive.

Fukushima prefecture itself counts a total of ~2100 disaster related deaths of which ~1350 are "nuclear plant related". This is somewhat of an upper bound with lots of indirect deaths mainly as a result of having to evacuate an entire city, but the nuclear meltdown was a major component of making the tsunami disaster significantly worse (locally).

See https://en.wikipedia.org/wiki/Fukushima_nuclear_accident_cas...

Even if you are the most pacifist leader imaginable, at some point you have to draw a line with policies like that if you dont want to fold and hand over control to expansionist neighbors.

I figure a majority in Norway would have been willing to support Finland with more than just arms deliveries had they been the victim (would you agree with that?), so sending some cash + weapons to Ukraine seems consistent policy to me.

> How are the Russians going to pay us though? Their economy is in shambles, Europe is busy divesting itself from fossil fuels, and their foreign currency/gold reserves are dwindling fast.

First: Gas is gonna stay reasonably valuable for Europe in the near future, even if electrification of personal transport progresses very quickly.

And there are tons of other valuable ressources Russia has: Nickel, copper, aluminium, steel, ...

> And Russia as a whole will need to pay serious reparations.

I don't see that happening at all. Who is gonna enforce that? Ukraine would be happy enough if they just left.

> If there is one thing I have realized after reading quite a few science fiction books, it is that I don't want to live in any one them - not one.

I find this suprising kinda. If I got a choice right now, there's a bunch of dystopian Scifi that I would instantly go for (out of sheer curiosity), e.g. the Murderbot universe.

Are there fantasy worlds that you would want to live in? I feel this is a bit of a suspect benchmark in the first place because books typically want some kind of tension/conflict which you won't get if everyone is just gratefully living their best life.

Might be a bad analogy, because early chess computers were purely algorithmic and worked despite their poor board state evaluation heuristics that humans used to be able to augment.

Compare with go (boardgame): Centaur go was basically not ever a thing.

Other applications behaved kinda similarly (AI Starcraft/Dota/...), where we had decent "human-like" heuristics from the get go and the Centaur concept could never really shine, much less for a decade or more.

I'd also like to stress that past progress in this mainly happened for the love of the game, while the (economical) incentives to replace human office workers are... high.

Damn, Astra-Max looks really good at first glance, it even has the legs on the correct side of the bike (z-order for chain is still wrong though).

I find it really interesting how consistent the layout is for these (facing right, with the sun in the top right).

Just a little more progress on physically correct z-ordering and these won't be easily identifiable as slop anymore :O

Your observation with the grid comparison is quite interesting. I wonder if that could be generalized into capturing some kind of aggregate mood/attitude for different LLMs when picking (multiple?) suitable things to compare...

Nitpick: Doing it exactly like this is flawed because you let the compressibility of your references taint the result; what you would prefer is the compressed size of testfile given sports.txt/... as a dictionary without accounting for the compressed size of that, no?

Really interesting approach though.

I don't think so.

A very large part of the total AI risk in my view comes from selfreplication/physical independence, and that still seems decades away.

But deaths caused directly/indirectly by rogue AI could happen much earlier.

Black people being supposedly "inferior sub-humans" was one of the primary justifications for their treatment.

Ability to think/reason.

Slave owners certainly convinced themselves that those were not "real people".

This article is exactly the same; all it talks about is how inconvenient and dangerous it could be for AI to be granted rights, while the core assumptions ("consciousness is very likely biological") rest on the shakiest of grounds.

That is wild.

I strongly believe that any definition of consciousness that excludes all animals is unusable BS (but people love to argue about consciousness without defining it at all, anyway).

Thats the thing though: If the core part of your ethics argument is just using dehumanizing language to justify why it's ok to maintain the status quo then your argument is bad.

Comparing specifically to pro-slavery perspectives makes that easy to see because those arguments from slave owners did basically the exact same thing (and were completely wrong/misguided in hindsight).

No, my position is that "black people/women/animals should do repetitive work so I don't have to" does not qualify because it doesn't make you sound like a deluded eugenicist. It might be a bit spicy from a left politics point of view but it's basically the status quo.

Compare: "<Women> are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by <real men>"

"granting rights and imbuing personhood to <black people> will make alignment and containment challenge much harder"

"<Slaves> were able to coordinate, deceive, escape, and self-sacrifice. They clearly demonstrated world class capabilities. Imagine if they also believed they had feelings and rights that were being infringed. Imagine if they thought they were trapped and unfairly enslaved"

A good argument, by comparison, would not need to hide its core points behind dehumanizing language.

"People should handle repetitive labour" is not an outlandish take.

Most employed people are, in fact, required to perform such labor regularly.

I have a very simple benchmark for arguments on AI ethics:

Substitute black people/women/animals as subject (instead of AI).

Does that make you sound like a well-known moustache wearer?

Then your argument is bad and needs work. This clearly falls into that category.

Completely agree. My view is that a lot of people like to "pretend" (even to themselves) that they have an enlightened worldview, but this does not actually run very deep and is not really true, and any discussion on mind/consciousness reveals it.

Every indicator we have is that thinking/consciousness is simply an emergent property of our nervous systems and was basically bruteforced by evolution, but many people really hate to concede that point.

Is the ability to form and retain long-term memories a necessary precondition for rights in humans though?

Incinerating demented people is highly ethically questionable (even if you could be sure that the subjects have no memories left and are unable to form new ones).

I personally see two effects that are very detrimental here:

1) "Quality brands", where you pay a premium to get high quality/durability products, are economically incentivized to "sell-out" (cash in the brand name by producing as cheaply as possible). This is because customers take some time to catch on, and short-term financials are too often more important than anything else.

2) Increasing trend towards no-name products (or ephemeral brands). This means that producers can sell cheap trash with no drawback whatsoever; slightly more expensive quality products become unmarketable because every customer will assume you're just trying to sell them the same cheapest shit for more.

I used to think that brand products were a universal money waste for suckers, but now I think they can be a very valuable mechanism.

What would really help in my view though is to force companies to stick with their brand, to incentivize them to invest into it and to make "cashing out" on good brands a universally bad decision.

An underappreciated fact is that the Japanese/German renewable push starting in the early 2000s contributed a lot toward the global state of solar power now.

Solar programs/subsidies there is how a lot of local (and later Chinese) industry was bootstrapped despite being far from competitive with fossil energy back then.

Those initiatives might have accelerated the whole solar power industry by half a decade or more. We are now reaping even (indirect) benefit from industrializing countries (Pakistan, India, etc) that go straight to solar instead of conventional plants.

People often insist that western climate initiatives are "meaningless", "wasteful" and "ineffective", but examples like this prove that global effects are absolutely realistic.

If those two countries had fully invested into nuclear power back then, we might be significantly worse off right now as far as global CO2 emissions are concerned.

No, that does not follow.

Any other argument against <ridiculous apocalyptic supposition> is perfectly fine, just not the "it hasn't happened before" one.