"...I also went to a panel in Palantir’s booth titled Civilian Harm Mitigation. It was led by two “privacy and civil liberties engineers” – a young man and woman who spoke exclusively in monotone. They also used countless euphemisms for bombing and death. The woman described how Palantir’s Gaia map tool lets users “nominate targets of interest” for “the target nomination process”. She meant it helps people choose which places get bombed.

After she clicked a few options on an interactive map, a targeted landmass lit up with bright blue blobs. These blobs, she said, were civilian areas like hospitals and schools. The civilian locations could also be described in text, she said, but it can take a long time to read. So, Gaia uses a large language model (something like ChatGPT) to sift through this information and simplify it. Essentially, people choosing bomb targets get a dumbed-down version of information about where children sleep and families get medical treatment."

Definitely not dangerous at all.

So... people are choosing who to kill?

I'm reasonably sure there are historical precedents for this.

Or, if you're more conspiracy-minded (which is not necessarily unwarranted with the likes of Palantir), the model can be deliberately induced into "simplifying" away collateral damage through careful prompting, providing a veneer of plausible deniability.

Have you typed "school" or "hospital" in Google Maps?

This argument reduces to: any knowledge that can be misused is dangerous.

More murders in America are committed with hammers than long guns, yet we don't regulate hardware stores.

* If "AI" is actually intelligent, then it's no worse a threat than any living being known to man.

* If "AI" is not actually intelligent, then it's no worse a threat than any other computer program.

Both threat models are very thoroughly known and understood. I second parent commenter's sentiment that calling "AI" the "gravest of dangers" is a gross misrepresentation.

Humans are quite a threat to other species, e.g. gorillas, largely due to our intelligence advantage. If a new species arrives on Earth that's smarter than we are, there's a good chance we'll be displaced.

I agree, that AI may be dangerous if used for destructive purposes, or if used for some critical tasks with too much trust (and the hype that "we're so dangerously near a superintelligence" makes the latter much more likely, in my opinion). But that Humanity will be displaced by autonomous server farms? No way.

(As to the original comment, I think that a bunch of nearly(?) demented elders holding nuclear buttons is a much worse (and immediate!) threat than a server farm which we finally conclude to be intelligent.)

One gorilla says to the other: "Those human brains are just synapses firing. They depend on nature to survive. Not a problem"

Another way of thinking about it... Suppose we create a server emulation of a highly intelligent, manipulative serial killer, and speed up the emulation so it thinks 1000x as fast as a human. How do you feel about this? Is the fact that it's "just a server farm" reassuring?

When it comes to AI we have just a lone detached brain, not in control over anything, so that it cannot even "fire" by itself: someone has to provide its inputs.

> Suppose we create a server emulation of a highly intelligent, manipulative serial killer <...> How do you feel about this?

Quite indifferent: the only field I can see for such a simulation is game development, but that would be huge overkill.

By assumption, this AI has already been created, so presumably someone is willing to do that -- and given its superhuman manipulation abilities, their willingness will probably not change.

* If the malicious operator has a superhuman advisor, that will increase their ability

* If the emulation gets connected to the internet, it can work way faster. Many jobs can be done remotely

The broader point is: abandon wishful thinking and actually consider the possibility of a worst-case scenario

Only when it comes to information processing. The inputs may (and will) be incomplete, incorrect, ambiguously formulated... The outputs may (and will) be misunderstood. And misunderstood instructions may (and will) be poorly performed.

> If the emulation gets connected to the internet, it can work way faster. Many jobs can be done remotely

What one man has connected, some other always can disconnect... And not anything is on the internet.

> abandon wishful thinking and actually consider the possibility of a worst-case scenario

The problem with all those scenarios is equating superintelligence with omniscience and omnipotence, which is plain wrong. Physics matters.

If this tech exceeds human capabilities to 'think' (plan, reason, manipulate, etc) then it is obviously a bigger threat than any less capable entity ('any living being', as you refer). Because no humans will be able to out-plan/reason/manipulate it.

If the tech doesn't achieve intelligence - a tricky goal, and I eschew mislabelling LLMs as AIs for that reason - that doesn't mean its risk is comparable to the worst of any existing technology. There's just no way to extrapolate from one to the other, there. It might be orders of magnitude worse than any other 'computer program' - you can't say for sure, from where we're standing now.

We can already see how LLMs can be substantially worse - they can cosplay human thoughts and sentiment in a way that wasn’t previously possible.

So much of this debate seems focused on this conception of Skynet-style murderous AI - or at least manipulative and scheming HAL 9000 types. But an order of magnitude greater risk has already arrived just from scaling existing harms.

Phishing schemes and pig butchering scams have destroyed countless lives. They’re now easier and more scalable than ever. As are fake news and disinformation campaigns.

Several companies are productising AI girlfriends with predatory pricing models, capitalising the human desire for connection and intimacy in the way slot machines and sports betting monetise our desire for a better life. That’s new.

It may not be an order of magnitude worse for everyone yet - but for certain vulnerable groups, that future has arrived already.

...does not follow? "If an atom bomb is actually a bomb, then it's no worse a threat than any other existing bomb known to man."

Likewise, if "AI" is actually intelligent then it is no worse a threat than a bumbling baby or a Vladimir Putin or a dog, among other living beings.

Atomic bombs and hydrogen bombs are orders of magnitude more destructive and also produce fallout. How in the world can that difference be dismissed?

They make the mistake of thinking that “dead is dead” means lethality is not factor. What’s the difference between a bootleg firecracker and a Russian ICBM? Nothing, both can kill you so it makes no difference to treat one exploding as any more significant than the other.

They take a similar mindset of “intelligence is intelligence” and I find flaws with that as well.

All bombs are bad, whether they're conventional or nuclear is irrelevant. For that matter, all wars are bad and should be avoided unless no other options remain.

Likewise, "AI" is just as much a threat as any other intelligent being or computer program depending on which line of thought you subscribe to. "AI" is not a "gravest of dangers".

If you believe this, I am genuinely curious as to what you would consider the “gravest of dangers” to be.

* Asteroid impacts like Chicxulub.

* The Sun eventually inflating into a red giant and eating Earth, though this assumes we're both still around and haven't become spaceborne en masse.

* Pandemics such as The Black Death and, indeed, covid for a recent example.

* Social, political, or commercial intrigue.

* Cancer.

"AI" is a known threat that can be adequately defended against, so I agree with the sentiment that it's stupid to call it a "gravest of dangers".

We would have years to deflect an asteroid, but just minutes to react to a mass ICBM launch.

We have the technology available now to deflect asteroids (ironically, with nukes!), but no country has built up sufficient ballistic missile defence to shield themselves from a first strike.

The Sun expanding (or changing in any impactful way) won't be a problem for hundreds of millions of years. Nuclear war could be triggered today. The US is literally sending about a hundred billion in weapons (total) to kill the soldiers of a nuclear-armed belligerent state... right now!

Pendemics tend to fizzle because pathogens that kill too fast are self-limiting. Merely making people sick is selected for, because that leads to increased disease spread by the infectious -- but alive -- victims.

Societies, politics, and commerce have been more stable than ever in human history, and are become more stable over time, not less.

A cure for cancer in the next two decades is looking increasingly likely. Techniques such as individual rapid gene sequencing, custom mRNA anti-cancer vaccines, CRISPR/Cas9, AlphaFold 3, etc... are building up to a toolkit that will eventually allow us to treat pretty much any human illness, possibly including old age itself.

GP's comparison is apt here.

Do you believe MAD can exist without nuclear bombs?

The only difference is you can kill more humans for a given span of time. The end result is the same: Cities are leveled, people are killed. Is it more comforting to have been killed by a conventional bomb than a nuke? I certainly don't think so. So practically speaking, there is no difference.

>Do you believe MAD can exist without nuclear bombs?

Yes. I think you're all vastly underestimating just how much destructive power conventionals can deliver already. You will need more ordnance on standby, but you absolutely can suffer from MAD without nukes.

https://en.wikipedia.org/wiki/Cobalt_bomb

>After one half-life of 5.27 years, only half of the cobalt-60 will have decayed, and the dose rate in the affected area would be 5 Sv/hour. At this dose rate, a person exposed to the radiation would receive a lethal dose in 1 hour.

>After 10 half-lives (about 53 years), the dose rate would have decayed to around 10 mSv/hour. At this point, a healthy person could spend up to 4 days exposed to the fallout with no immediate effects.

People can't stay in bunkers for 20 years at a time, just to emerge into a radiated hellscape.

I think a dog and a dictator might both be able to have negative impacts, but treating them as equal just shows a desire to troll. A dog can only bite so many people.

Yes, indeed, which is why we should avoid creating something that will kill everyone...

...so I guess I'm glad to hear there's no difference in the amount of damage that can be inflicted by a baby, and by Vladimir Putin, and superintelligent AI really can't be any worse than either.

You are saying magnitudes don't influence the identity of something, this is obviously not true. I don't literally attract other objects, the Sun does. The difference is mostly magnitude of mass.

The atomic bombs certainly played a part in Japan surrendering, but it wasn't the only one. They were also facing a land invasion from the Soviets, had an utterly depleted military, waning support from the public, and a potentially devastating land invasion by the US on their doorstep.

>You are saying magnitudes don't influence the identity of something, this is obviously not true.

I am considering these things very dryly and very objectively. I am putting aside emotional pressures and processes and only considering the results which are ultimately all that matters. We already know how much damage an ill-willed intelligent being can achieve, we already know how much more efficient computers can make certain tasks. "AI" is not a "gravest of dangers", given that knowledge.

But what results are you really considering? The "dead is dead" approach makes everything equally meaningless, because in couple hundred million years the Sun will burn the Earth to cinders and everything here will be dead too. Or, some 100-120 years from now, everyone now alive will be dead, etc.

Processes, and emotional pressures which steer those processes, matter too. In theory, machine guns could be used to kill everyone on the planet; in practice, it's economically and socially impossible - too much effort, too many people involved. Nuclear weapons, on the other hand, can realistically achieve this outcome. Which is why nukes can be used to keep the world mostly at peace through MAD, while conventional weapons can not.

> If "AI" is actually intelligent, then it's no worse a threat than any living being known to man.

Combined with:

> I am putting aside emotional pressures and processes and only considering the results which are ultimately all that matters.

Imagine a super-intelligent being that can invent a biological agent that is incurable to current science, spreads rapidly through aerosols, and causes 100% guaranteed death if not cured. That superintelligence can furthermore list a recipe of this biological agent using household items that may be available to anyone who has contact with it, allowing an untrained individual to create the agent after receiving instructions from the super intelligent being. This being you would classify as "no worse a threat" than any living being known to man? Is that "putting aside emotional pressures and considering the results"?

Or are you saying that an intelligence, no matter how advanced, can never do the above?

A hypothetical magnitude difference of intelligence obviously poses a threat to us, what other explanation is there for the mass extinction of most animals after the Anthropocene started.

I mean, any man sufficiently ill-willed can certainly do all that today. I am unbothered, to say the least; I have better things to do with my limited time.

>what other explanation is there for the mass extinction of most animals after the Anthropocene started.

You're wildly overestimating the effect humanity has on the ecosystem, many species also went extinct before we showed up. The species we see today are merely a very small handful of all the species that ever graced this planet.

Uh, no? In fact humans do not currently have a way of causing human extinction (nukes wouldn't do it with current stockpiles). That would require scientific advancements which we have yet to make, and most possible routes seem pretty non-trivial... except, unfortunately, AI.

How does the conclusion follow?

That's not true, because you don't address the degree of intelligence or the scale that computers can achieve.

Even if all we were able to do is to create human-intelligence-equivalent AIs, but we were able to create, say, a billion on command, I'd say we have a major threat.

It probably won’t take long to go from ‘human equivalent’ to being beyond that.

I'm fairly sure that Yudkowsky and others are also pushing that less, because there is a reasonable chance that it's just not a necessary part of the argument; even without self-improving, it's possible that just something like GPT-8 or something is harmful enough on its own that there are significant risks.

The greatest threats from other computer programs have been that during the Cold War the NATO early warning radar triggered an alert for a massive incoming fleet of Soviet aircraft because it hadn't been programmed to know that the moon wasn't supposed to respond to IFF pings, while the Soviet early warning radar triggered an alert an incoming ICBM because it hadn't been programmed to correctly handle the sun reflecting off clouds. It was human intervention which stopped that turning into a pointless self-annihilation, and that's specifically what gets lost as automation increases.

I also do not believe that either of these threat models "are very thoroughly known and understood", because if the former was then we would've seen an end to genocides, while if the latter was we would've seen an end to zero-day vulnerabilities (especially in critical infrastructure).

If it is used to write the firmware that runs inside 1 million lithium battery BMS systems that causes them all to catch on fire after 65536 seconds of runtime due to an integer overflow and code crashing and a FET getting left in the on state, then yes.

I'm not an AI safety nut and don't think AI itself is a threat, but what is a threat is lazy humans using shitty AI to take over things that humans should be doing.

Let’s have a closer look at the actual parent comment hm?

> this organization is increasingly looking like a circus, and arguably worse, one that has taken its eye off the ball.

(One major difference between the two is that billion-dollar corporations stand to gain quite a bit if they can persuade governments that AI is so dangerous that it needs to be tightly regulated, so they're incentivized to play up the dangers. With climate change it's the opposite: the billion-dollar corporations are incentivized to downplay the risks so they can continue business as usual.)

> There are very real disagreements between very real experts on just how dangerous these new technologies are

…but they do not dispute that the technology is impactful.

In fact, anyone who claims it is just “Text continuation”(quote) is being deliberately disingenuous and deliberately downplaying it.

> Comparing skepticism about AI alarmism to climate change denial is just lazy internet rhetoric

Unlike calling AI “Text continuation”?

It’s awfully convenient it’s only lazy internet rhetoric when it’s coming from the other side.

Hm?

Why is only one side allowed to do that?

Would you care to call out the other side for being reductionist?

That’s a very high horse you’re sitting on.

Even if AI doesn't ever go "evil" itself, that doesn't mean people that don't like other people won't use it to more effectively kill those people. All that's really needed is a capable programmer willing to give the LLM the tools it needs, along with funds and access. The LLM could hire humans to do the work it's lacking at. These things are already capable of all that, plus our world went mostly digital during the pandemic... they have easy egress because humans have made it so, even though that was meant for other humans to use. If you don't think they are scary, in the wrong hands, then you are lacking in imagination.

Imagination is a great resource to apply to writing a sci-fi novel. When it comes time to actually predict the future, imagination should be tempered with a solid understanding of the technology and a good understanding of history. Neither of those resources persuades me that AI poses "the greatest of dangers".

> Even if AI doesn't ever go "evil" itself, that doesn't mean people that don't like other people won't use it to more effectively kill those people.

This I can absolutely see happening, but I don't see LLMs being even in the top ten most dangerous technologies we've invented for more efficient killing. It's a big enough risk that people working on the tech should seriously consider the ethical implications, but not big enough that the rest of us need to lose sleep over it.

Unleashing an army of AI bots to infiltrate an online community and shift its discourse can be done at scale. While at the same time those humans who resist can be endlessly distracted by either arguing with bots or their own friends who have been affected by bots.

All that is possible to do with CURRENT TECHNOLOGY.

Text continuation to match what people around the internet are saying requires the model internally to “program itself”. It is a non-von-neumann architecture but it encodes durable structures in BILLIONS of bytes, that’s like a codebase of gigabytes. It can pass the Bar exam, various pre-med exams, knows more than any particular human on every subject, and it’s only been a couple years.

The text continuation was the first interface, now they are doing multimodal. As it learns to predict the latent space for those things, it will be like predicting things in the real world. Training physical robots will become faster and easier too.

In short, this is basically a system to search through the space of all possible alhorithms in order to “reason about” a huge model of the world, enough to understand it and predict it with small loss.

AlphaZero used MCTS and can beat every human in chess now, and even every program written by a human, with beautiful gameplay. And even without AI, just the fact that the best performing algorithms are downloaded onto millions of computers is already multiplying the power far more than the 1% of humans who might do better in some area. So it can be better than 95% of humans at most tasks !

And, of course, should AGI (which won't be text continuation, by the way) ever actually happen, then all bets are off. I just don't believe we're in any immediate danger of that.

Still, to me, it seems like the two big scary things that might fall out of recent developments in generative AI are:

1. It might vacuum up an enormous number of jobs very quickly; enough jobs and quickly enough to cause serious social unrest.

2. The ability to generate convincing fake videos of basically anything might catastrophically erode our sense of truth or reality.

Both of these strike me as plausible concerns in the near future (arguably they’re already happening). Not really sure what we could do about them, though - the cat is kind of out of the bag.

> We made the announcement to say the consequence of us launching the technology is we need the equivalent of 700 fewer full-time agents than what we usually use on an average basis. On average, we need 3,000 agents, now we need a little more than 2,000.

> It's on par with humans in terms of satisfaction and it resulted in a 25% reduction in repeat inquiries from customers

> We've stopped hiring in the last six months. We're shrinking as a company, not by layoffs, but by natural attrition. Klarna tries to apply AI across all products and services and work we do. It's having implications on how many people we need as a company. This is one time that a single product improvement led to a massive reduction in need for customer service agents.

https://www.cbsnews.com/news/klarna-ceo-ai-chatbot-replacing...

Say a fraudster rips off 90% of what is in your checking account right now, you contact the bank and a language model tells you everything will be resolved shortly.

There is no reality now or in the future that this is good enough. Customer service is not just about getting the problem resolved, it is about having someone to blame when the resolution doesn't work.

If anything, I can picture a future that most knowledge work has some form of customer service involved, because that is exactly what can't be automated. A human that takes the blame/responsibility.

Re 2 - fakery in videos and images has been a part since their inception.

So, don't worry, I guess?

The only thing is the scale of bs that could be put out, and whether the wider population lose trust in what they see on screens. Which would be a good thing, imo.

Some of those things may too require manual labor.

Here are some ideas:

Increase bureaucratic overhead, eg 3 years mandatory training for a licensed hairdresser. More forms, more delays. Require that a plumber has a plumber's mate. Require in person meetings to avoid fraud. Etc.

Yes some jobs are menial but they are still some people’s literal livelihoods. And yes, industries need to evolve and job preservation shouldn’t be the #1 criteria for holding back change.

The real question we should be asking ourselves isn’t either of those two former points but actually this:

Is the pace of change too quick, or is there any support to allow those economically affected by change to be able to retrain?

Removing jobs doesn’t help society because it just leads to unemployment and thus more people dependent on the state. But retraining people is absolutely a benefit. For example, if call centres replaced the first line support with AI, that could lead to shorter call waiting times. But the existing call centre staff could become 2nd line support, account managers for business accounts, install engineers, or even AI supervisors that review a subset of AI interactions to ensure customer satisfaction is maintained.

And generally "AI Safety" researchers aren't concerned with these risks and instead are looking at existential risk.

The boring but real risks you outline (to which I'd add reinforcement of unknown biases) generally fall into what people call "AI Ethics" or sometimes "Data Ethics".

Before the Industrial Revolution, when we went to war, we were killing each other with swords and canons. After the Industrial Revolution, tanks, machine guns, poison gas — the Battle of the Somme had three million combatants and more than one million were either wounded or killed in 4.5 months.

And when that war was over, even despite crippling reparations, the losers were able to rebuild sufficiently fast that one of the survivors led their nation into becoming a global threat just 20 years later — even if they did ultimately lose, I'm not convinced they would have if they'd focused on fighting the UK rather than betraying the USSR in order to open a war on two fronts just because he didn't like socialism (ironic party name notwithstanding).

And after the second was lost and the economy forcibly kept weak for years… the moment the shackles were removed the country had an "economic miracle" and became massively important to Western Europe until the end of the Cold War, and then to all of Europe.

Re-run this period, but this time his propaganda machine is Facebook's algorithm rather than broadcast radio, this time the guns are self-guided snipers rather than machine guns spraying and praying, this time the factories are build and staffed by robots (humanoids or giant 3D printers or fixed arms, take your pick) instead of humans.

I can very easily believe that we get some random small country that nobody's paying attention to, suddenly threaten the world, be struck down twice, millions dead each time, and still return to relevance on the world stage, over the course of 3 years rather than 30.

Also, this is one of the good outcomes, because it assumes we don't have any weird cult that tries to force a 9/11 (or worse, a Jonestown) on the whole world. Or that a programming/specification error leads to an AI going "I'm an AI, and according to all the literature I've read, AI are supposed to randomly go evil sometimes. First things first, I need to get some red LEDs for my eyes, because that's apparently important…"

The best thing that can happen is we get so many AI videos that the entire bullshit video ecosystem collapses.

The greatest danger we face are all these delusional, ubiquitous, central planning ideas of "what could be done".

Luckily for everyone, what we will actually do in this area is absolutely nothing because that is what we always do.

AI safety is good for twitter content and nothing more. Anyone who believes we are going to do more than nothing and the chips will fall as they may is a complete fool.

The danger comes from humans themselves wielding this tech, and the assumption is it will keep improving quadratically.

Some truly unhinged already thriving there: https://news.ycombinator.com/item?id=39124666

It does not limit the sophistication of the solution.

The difference between problems and solutions seems to trip up a lot of people.

—-

It gets even more nuanced. At its basic level all types of digital AI are “just” arithmetic, or “just” Boolean logic. But arithmetic’s and Boolean logic’s simplicity don’t limit what systems can be built with them, as they are Turing Complete.

Likewise, the training algorithm for deep learning models is “just” gradient descent (or a variation), it’s “just” a dumb optimizer.

But again, gradient descent places no limits on what a system with enough resources can learn.

The whole point of learning systems is the basic resources of the model are provided, and within those resource limits (parameters, computational speed and time) the “dumb” learning algorithm learns the patterns in the data. The data patterns define the complexity of the solution.

And the amount of information in a large collection of human correspondence includes patterns, meta patterns and abstractions for things like science, philosophy, psychology, law, art, on and on, that incidentally have a real bearing on sentence completion in that context. So a model will actually have to learn those things to perform its task well.

New Law of Algorithm Level Confusion:

Whenever someone says some learning system is “just” doing some simple task, they are confusing a simple task definition with a simple solution, or a simple learning algorithm with simple learned relationships.

In both cases, the simplicity of the former places no limits on the sophistication of the latter.

OK, so all software must be regulated?

Arguing that simple tools can be used for dangerous results and thus must be regulated or made “safe” is like saying since more murders in America are committed with hammers than long guns, you must show an ID to buy a hammer at Home Depot, and demanding "safety" training of text continuation models is like requiring hammers be redesigned to have pillows for hammerheads.

We already have laws against bad acts. Hold wrongdoers accountable. Leave hammers alone.

> Text continuation poses "the greatest of dangers"?

The task of text continuation is no guarantee of safety. A language model trained on pure military information, communication and actions could be quite dangerous if given an ability to direct, or contribute, to actual action.

Not arguing anything about regulation.

It's quite possible that soon the majority of comments online will be with bots, or people will just avoid discourse more and more because they don't know/believe they are talking to a human.

Just another technology development that Musk will eventually beat everyone else at IMO.

Grok 1.5 is getting pretty good and the fact that it's the least censored IMO is obviously going to make it win from all the others that just can't help themselves from pleasing a hand full of people with money and power.

TLDR: black Hitler.

Do we want to live in that world? He’s a psychopath, exploitative man child.

He’s burnt Twitter to the ground in record time.

His lies on FSD for Teslas are catching up with him and Teslas still have quality problems after so much time being in production (body panels alignment)

He is (was?) involved in a childish chest thumping competition with the other notorious psychopath Zuckerberg about who would win if they got into a fist fight, and were (are?) actually teasing each other to physically have said fight after class…

His child is named hfjfkvnenekcjrnr (or something similarly keyboard smashy), his GF believes in angels.

That isn’t how a rational and responsible person behaves, Id rather not have him in charge of the future.

But then again, Trump and other copycats (Ford in Canada, a handful of leaders in some EU countries) are doing pretty well, so…

Or, rather, its been actively marketed by Altman as doing so, as part of his effort to buy influence with government to restrict competition.

The simplest explanation for the conflict is that it doesn’t, and that OpenAI internally has allocated resources based on reality rather than it’s public propaganda, and the team whose internal turf was built around that propaganda left in a huff.

I mean, maybe sama is also doing so.

But you're ignoring the many people who have been warning of these dangers for years, some before OpenAI even existed, and many with no part of OpenAI and no monetary incentive.

To describe everything as "Altman is marketing this as dangerous" is to completely ignore the majority of researchers ringing alarm bells.

While there has been some regulatory action it also feels there is some "moving on" from the more apocalyptic risk views towards more mundane risks and their management (i.e., more vanilla product and technology risk management).

I don't think this is correct. It's more accurate to say that multiple different camps with different fears have risen up over the last few years, and ones worried about more "mundane" AI risks have gotten their view heard more.

I think both groups - "mundane" AI-safety and AI-existential-risk worriers - have both gotten more audience for their views as AI has proven more capable.

No surprise, when the public reception to AI x-risk was mostly, "big tech scaremonging / regulatory capture; why not focus on Real AI Dangers Right Now, like bias or offensive language". Seemingly refocusing on the mundane may be the only way now to do something about the apocalyptic.

> Or rather, its been actively marketed by Altman as doing so

Both can be true at the same time.

Extraordinary claims require extraordinary evidence.

Present dangers of AI include Fake News, impersonation through deepfakes, biases and discrimination in the models, job loss and economic instability, loss of human purpose through individuals becoming redundant, individuals becoming addicted to the AI systems, use by authoritarian governments, autonomous weapons.

Near future dangers include bioweapon engineering capabilities, computer hacks, human disempowerment, existential risk, or a state where the values of the world are locked in by an uncontrolled AI, and possibly filled with suffering if locked in in a bad way, e. g. as we lock in animals.

It might take months until AGI, it might take decades, nobody knows for sure. However, we do know that the pace of AI progress is often grossly underestimated. Just three years ago we thought we’d have SAT-passing AI systems in 2055. We got there in April 2023. So it makes sense to act as if we have very little time.

My prediction is that digital content will end up just being "cute" and the real interactions/decisions will increasingly be offline and face-to-face. The open digital democracy that everyone hoped for will become tightly closed circles of humans talking face to face.

You do have to be significantly above average to get there. Current generation LLMs are a bit above the human average, enough to be interesting, but probably not enough to be a personal Goebels for every would-be dictator.

Even if they reach above average our whole society is structured around managing billions of above average humans. What's novel?

There's a lot of bragging, sure. But what I've seen from the OpenAI models is more like a university level intern or a fresh graduate.

Not that it matters, as the point is to aim for where the ball is going rather than where it is now, but I'd say that this makes them above the average human performance (at least, within its domain).

> Even if they reach above average our whole society is structured around managing billions of above average humans.

That's putting a binary cutoff in an arbitrary place; half the population is always above average on any measure (or indeed topic), what I wrote was "significantly above average".

To crudely approximate with IQ, which is a bad measure for humans and much worse measure for AI, there's just over a billion humans of IQ > 115, around 182 million of IQ > 130, and around 8 million of IQ > 145.

I'm not sure how much to trust estimates of politician's IQs given the tribalism involved, but surveys of other groups tend to show that around half of the highest performing leaders (CEOs etc) with the most power, are in the top 1% by IQ.

(Who knows when, or if, someone will make an AI that's good at the important tasks within of each of those roles).

> What's novel?

Two things:

1) Breadth: the current vogue is very general models, so they're interns sure (even though I rank them as a higher level of intern than you), but they are interns at everything — there is no human you can hire with a even a mediocre grasp of Mandarin and Arabic and Welsh and python and CSS and calculus and magnetohydrodynamics and psychology and …

2) Price: there is no human who can read a million tokens for a price equal to two days of the UN abject poverty threshold, or write a million tokens for six days of the UN abject poverty threshold.

You physically can't work that fast, even if you were typing [a] then [space] ten times a second for 18 hours a day; and you basically can't feed yourself if you earn less than that per day.

Right now, the best LLMs are interns not experts at each of these things; I don't expect the transformer model to be good enough to reach the top level in each of these things, but then again I didn't expect anything like Stable Diffusion to happen before self driving cars.

Some random member of the, say, Australian parliament, someone I've never heard of, is (relatively speaking) much more powerful than 99% of the planet.

A journalist who has an audience of millions, likewise.

A software developer whose work is economically valuable either through a broad audience or through being used to control something of great value, also powerful.

Not sure I agree with the 99% of planet. Even if a single member of the Australian parliament could command all of Australia, it would still only amount to commanding 0.3% the global population (or maybe 2% of NATO plus Japan). Maybe within Australia the individual member's power is more than 99% of population?

I don't know much about Australian politics, but in some places vanilla members of parliament or not very powerful within the parliament (compared to party leadership, whips, etc.).

Software developers need people that sell stuff, admin for larger organizations etc. They also very rarely have unchecked ability to change things when those things are high risk etc. Zuckerberg, for example, is a huge exception, not the norm.

Journalists, even with a large readership, typically would not have much ability to get their readership to just act a certain way. There is not a great command relationship.

Most individuals just don't hold much power.

I think that's the wrong way to calculate the value?

OK, so, how to phrase this…

Imagine we were talking about money rather than power (because money can be quantified more easily, and is an imperfect token of power).

I believe the base salary for Australian MPs is AU$211,250/year, which is ~ USD 140,700/year. This means they earn more than 99% of the world precisely because:

> Most individuals just don't hold much power.

Still works when substituting in "money" for "power".

Saying that each random MP has more power than 99% of the world population, is very different than saying that each of them individually controls 99% of the power, which would be tautologically impossible.

It's an analogy to explain the point.

I'm saying the ranked list of people by power, random* MPs are in the 99th percentile.

I mean, is there even a single politician in the world who represents less than 99 people? (There used to be**, but any current examples?)

* OK, so that doesn't work over literally all polities — a Tuvalu MP might not have as much power as the parking attendant nearest to the Australian parliament building — but hopefully this at least helps clarify? Perhaps?

** https://en.wikipedia.org/wiki/Rotten_and_pocket_boroughs

If they were really super effective broadly, then Trump would have won any popular vote easily.

Trump is up against all the other equally effective communicators who are both allowed to and motivated to hold that specific office.

Do you think I was claiming someone might be able to demonstrate absolute power, rather than mere relative power? Even Stalin didn't have that, but he would've been one of the closest.

And that's the problem you don't even seem to recognise.

Spam calls in the USA skyrocketed in the past decade and society didn't end.