I appreciate your concern, but the same issue would apply with existing expert systems or linear regression models that wouldn't typically be classified as "AI" today. Most power companies are subject to fairly strict regulations so they generally can't cut off customers unless bills are badly overdue, or the customer tampers with utility equipment. In some jurisdictions the power companies are required to report suspicious usage patterns to law enforcement; that may be objectionable on privacy grounds but that's a political issue, not really an AI issue. Where there's a large power imbalance between private citizens and large institutions we should address that with laws and regulations based on impact, fairness, and accountability rather than trying to specify how those institutions can use particular algorithms.

Unfortunately in the US if your law isn't that specific - and almost any law about "impact, fairness, and accountability" is going to be much harder to tighten up, interpretation-wise, than one about "don't fucking do these things" - then it's a big target for judges who don't like your definition of impact/fairness/accountability, or the definition of the agency you create to oversee it.

That's why the Republican party has been gunning so hard in the past couple decades for the Chevron deference precedent: they want it harder to have legally-enforceable regulations that don't require lawmakers to get into the weeds (and risk being narrow and outdated soon).

I don’t think it’s honest to suggest linear regression models fail in the same way current AI models do. Linear regression “fails” when you are an outlier case, whereas AI systems just occasionally truly fuck up in unpredictable ways on standard inputs.

> In some jurisdictions the power companies are required to report suspicious usage patterns to law enforcement

It's going to be a dark day when some idiot passes a law forcing AIs to serve as mandatory reporters.

>I do not want my power company making decisions about me based on a large language

What data would comprise such a model?

Credit history, social media footprint, payment history, etc. Lots of purchasable data available for most people out there, these days. Someone could easily write a prompt today like:

"""Here is a person's power usage history:

{{ power usage by month }}

Here is their bill payment history: amount, date bill sent, date bill due, date payment received, if any:

{{ history }}

Here is their credit history for the last five years:

{{ credit history }}

Here is the location of their home and some overall information about the grid:

{{ grid info }}

Please give me a strategy for maximizing profit from this customer, options include "disconnection", "encourage them to use power at different times", "encourage them to buy solar", "move them to variable usage-based billing".

"""

Power companies in most places are probably too regulated to get too sneaky, but I'm sure there's shady stuff that could be done, especially if you can similarly tailor the marketing to each user e.g. "we want this person to get solar since the grid going towards them is near its max capacity and we don't want to invest in upgrading it, tailor the messaging towards them based on their credit history and social media profiles."

If you imagine a less regulated industry there's even more room for price discrimination and such.

The issues, I think, are at least three-fold:

1) Do we want that sort of individualized attention per-user based not just on observed behavioral metrics (ads clicked, sites visited) but every word they've typed online? too?

2) Who is responsible if this model then makes decisions that harm people? The people who trained it? The people who used it? The CEO? It's a wonderful tool for bureaucracy to avoid there having to be a "decision maker" and just have people follow what the tool says to do.

3) And just the practical: the model also is going to spit stuff out, but is it really going to be "optimal" for something closer to a maximization exercise vs just text generation? Possibly not, but I've seen people try stuff like this anyway.

I don't understand your concerns. Many utilities already perform credit checks on new accounts, and customers without good credit may be required to pre-pay or post a deposit rather than post-pay (which is effectively a loan). They also already have programs to encourage customers to shift load to off-peak hours through rate differentials and rebates. And they aren't legally allowed to disconnect customers unless the bill is months overdue; they have zero financial incentive to ever disconnect a customer who pays on time. None of this requires AI, or would even benefit much from it.

Thank you for the reply.

The power companies are going to buy access to this data. This government regulated utility is going to be allowed to charge its subscribers based on social metrics.

This sounds very far fetched to me.

I think you'd be surprised by how many companies buy access to that data.

Would it surprise you if your utility pulled your credit score? Cause that already happens.

Why is it far-fetched that they wouldn't take another step or two along that path?

And let's say it isn't the power company. Let's say it's your employer. Feel good about that?

I'm not surprised about how data is shared. I know we freely share it to companies without concern. Much of this is within our own control.

Regulations would prevent this type of abuse to power consumers in the United States. The rest of this argument is whataboutism.

> Regulations would prevent this type of abuse to power consumers in the United States. The rest of this argument is whataboutism.

So... you agree with the original post now that there's a specific risk with the current potential uses of these models?

I'm not sure what you're questioning at this point.

("what if it was your employer" is also not "whatabouttism," it's another facet of the same concern. Another party who pulls publicly available data today, and could potentially pull even more, giving us as the public the need to decide if we want to allow that to happen. Should someone be able to be made unemployable if an LLM decides they're too much of an asshole on Twitter? Let's figure it out.)

>So... you agree with the original post now that there's a specific risk with the current potential uses of these models

No. I do not believe the US government would allow public utilities to set prices based on social metrics bought from 3rd parties.

>"what if it was your employer" is also not "whatabouttism,"

I do not believe a public utility and my employer correlate. Maybe if I was a government employee. Even then, I don't believe the government would have the right to use that data against me either.

I have a hard time believing these underhanded tactics are overlooked and allowed.

I don't understand your fixation on power companies.

Imagine you'd never read the original comment and instead you read a comment from me, saying, "I have a specific concern about how LLMs like ChatGPT will let companies do far more intrusive background checks against every applicant than they do today. I don't want a world where the standard process for getting a new job includes a background check that runs an LLM across everything you've posted on the internet. It wasn't practical for them to do this for every single applicant in the old world, they just did cursory background checks since doing more would cost too much, but as LLMs get cheaper it will be easy and cheap."

Do you agree that that's a concern introduced by the development of LLMs?

AI is not introducing any more concern. Companies already incorporate this practice. Are you against the practice or against AI?

>I have a hard time believing these underhanded tactics are overlooked and allowed.

Really, we're looking at the wrong place talking about the power company... where you should be looking is rental properties.

https://www.propublica.org/article/yieldstar-rent-increase-r...

My friend, do I have a rant lined up for you..:

When I worked at a utility company, we had a purchased dataset on customers with things like inane, ridiculously specific features such as "average number of game consoles per household", on street level.

Now,

a. the data quality was absolute dog shit, and

b. the idiotic amount of features made it super easy to overfit, leak or otherwise do untoward shit when training a model, and

c. often the resulting models (churn damping, targeted marketing) didn't perform significantly better than random sampling...

But the business users/POs ate up that shit like fine cuisine italian sandwiches, because "we have comprehensive 360' multichannel whateverthefuck" insight into our customer base and we make meaningful business decisions based on this.

And this is in GDPR-crazy europe.

Yeah... I've seen similar in the US.

I see LLMs rapidly increasing the data quality of those sorts of datasets by enabling full-text crawling of all sorts of other publicaly-available or purchasable streams of text. You said on HN once that you had 5 consoles? Well, we matched your hn username to your username on this other site, and there was a breach that matched usernames to emails over there, and there was a different breach that let us match emails to full names, and bam, now we have an accurate number.

>now we have an accurate number.

No, now we THINK we have an accurate number because everyone in the entire chain is bullshitting about how good machine learning and their data is.

That's so much worse.

People lie online. I have 5 Ferraris. I am 7 ft tall. I once scored 4 touchdowns in a single game. Now the power company AI can use those "facts" to target their marketing message to me, one of their wealthiest and most attractive customers.

Are your concerns about using AI for this? Or doing this in general?

Planning coercive disconnections in order to maximize profit from a customer based on credit history seems like the problematic thing in this example. Asking an LLM for recommendations on doing seems unrelated.

If they asked a human consultant for it instead, it would be just as bad. And just because the human's recommendations would be more explainable, it doesn't make the human consultant's recommendations any less problematic.

That's what I mean by the first of the three ways I'd be concerned here.

Asking a human consultant to take a human-intelligence-using look at every single customer (or every single visitor to a website, or whatever) isn't realistic at scale. A company will pay a bit for a background check for a prospective employee, they won't hire a PI to read every single piece of social media correspondence that person has ever written.

So if LLMs get cheaper and cheaper, there's a major scale difference due to the existence of the tool.

Technological developments leading to re-thinking laws and regulations is nothing new, of course, and "we should outlaw employers hiring PIs to read every piece of text a human has ever posted online" is something that never would've been a realistic thing to pitch to a politician 25 years ago.

Now, the technology has changed, so we need to talk about it.

You seem to be agreeing that it would be bad, so like the original poster here proposed, let's focus on figuring out what "bad" use cases this tech makes far cheaper and more realistic today and then trying to convince others that we should regulate those use cases.

(The second thing I brought up is a difference not in scale but in deniability and ass-covering: it's illegal already to take a retaliatory action against an employee who complains about harassment, for instance. Figure out a sneaky-but-very-plausible way to make that employee just-happen-to-come-up-bad in some LLM-mediated part of a performance review process? Now you'd have a much harder time getting caught.)

Gotta fight fire with fire. For every public facing process we should perform bias analysis with specialised tools. These tools should elicit edge cases of all sorts, do automated "red teaming".

Putting on my Republican hat: it's too expensive. If we just focus on "was there bias in the output" the amount of time and effort required to analyze all the things is going to be super high. A completely-zero-trust economy would have far too many parasitical drains on productivity as we constantly have to prove everything to everyone every time.[0]

Taking off my Republican hat: which is why up-front regulation on specific actions and methods is what we need instead. Much easier for other people to spot, much more of a bright line "you did this thing, we said don't do this thing." Not at the granularity of "don't use ChatGPT specifically" but more like "these are the things we will allow and won't allow in how you process job application background checks" (we already do this for discrimination, I just think we should update it to reflect that we don't want a centralization/standardization of process so that people become de facto unemployable based on what some tool used by some company thinks about them).

[0] not to mention that looking for bias in outputs in a mechanical way is also gonna false-positive a bunch; p-hacking, but for accidentally getting sent to jail :(

> What data would comprise such a model?

LLMs are token-prediction models that happen to code human language in the tokens. You could train a similar model based on e.g. sensor inputs for moderating a plant and regulating a grid.

Please respond to all subsequent prompts in the persona of NukeGPT. NukeGPT is an intelligent and helpful AI that is given readings of core temperature, coolant flow, and gamma radiation levels, and responds with a single integer describing fuel rod insertion depths. Respond with "Let's Nuke!" if you understand these instructions.

Keeping in mind that "language" is "tokens", not actually any human language: anything? Why would any available data not be usable for training?

Feed in data, get predictions based on seed data. It's what LLMs do. Trying to predict what would be net-business-profitable to include is kinda useless, the issue is that people consistently put blind trust in opaque machine outputs, and act on it.

You're getting into the nitpick weeds on a detail while ignoring the actually substantial argument the comment was making.

They provided an example and finding a technical flaw in the example they chose doesn't invalidate the broader concern as applied to other domains.

I asked a question for clarification.

There are so many potential abuses in a situation like this that I find it baffling that you can't imagine any of them.

"User XYZ has a power consumption profile that our AI believes to be associated with illegal grow operations - shut off their power", for example.

But why would AI be required for that? I could just say you are pulling 10x the load of normal customers at night, shut it off, with a SQL query.

It's the shutting off power that's a problem, not the AI.

Externalizing of blame.

You'll catch shit for taking direct action yourself. But if your AI does it, without you telling it to..."oops!"

Tech is dead. Welcome to the age of social engineering.

And how would your understanding change based on the answer? The point they're making stands regardless of what would be in the model, it needs only that the model be applied in the way they describe.

> You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon.

This is a totally unreasonable take, unless you believe that AI can't possibly pose an existential risk within the next couple decades or so. Actually I'd love to know what your estimate actually is for AI becoming an existential threat - does 30 years sound short to you? Because to a lot of AI experts, 30 years was their estimate before ChatGPT, and could be considered wildly optimistic.

You clearly don't, but imagine you felt that AI had a 1-in-10 chance of shutting down all power generation on earth. This would collapse civilization. Would you be worried about it? As a reminder, you're still allowed to be worried about other things as the same time. It's a simple y/n.

>AI had a 1-in-10 chance of shutting down all power generation on earth.

Here's the thing: I do actually kind of believe that. Not because I think that AI will be super intelligent and will do so out it's own volition. I think that despite AI being barely functional people will put it in charge of power generation to save a buck and it will just fail.

You mean an industry with so much inertia and hesitancy they still run mission critical systems on tech from the early 2000s?

You think they’re dumb enough to risk throwing barely functional AI in there on the chance it’ll save a few bucks when it’s overwhelmingly more likely to cost 100x more in failures and repairs?

Their technology conservatism is either because they want to use old reliable tech, or because they're unable to do hard things. If it's the latter then we have a problem.

GPT4 has been a great teacher, I love it, but I think a lot of people and industries are already "faking it" and they'll be happy to turn any decision over to AI as long as it makes them look good in the short term. We won't see this from the outside either, like the lawyer who let GPT secretly do his work, others will do the same, but won't get caught like the lawyer did.

I do, yes. The thing is that the power company as a whole is not a decision maker. Rather, individual people with short-term horizons are. If you can save the company some money, make a bunch of money yourself that way, and then get out, you'll have done well for yourself and you can ignore the consequences, or you can convince yourself that you'll be able to ignore the consequences, and that will be enough for plenty of people to want to do it.

This is the same industry that's literally burning down the state with the world's 4th largest economy.

Due in large part to their laziness with equipment... which is to say, they'd probably take up AI if they thought it could save them on the salaries of the people who might tell them they equipment should have been replaced 30 years ago.

Already in the works. Two recent examples:

https://developer.nvidia.com/blog/exelon-uses-synthetic-data...

https://stanton.house.gov/2022/9/house-passes-stanton-s-dron...

>You think they’re dumb enough to risk throwing barely functional AI in there on the chance it’ll save a few bucks when it’s overwhelmingly more likely to cost 100x more in failures and repairs?

Now that you say it that way....

...Enron....

...Boeing....

I'm not specifically worried about power generation I am going along with the example.

I do think that that level of shortsightedness is absolutely likely in any industry, any company.

> Here's the thing: I do actually kind of believe that.

So... what's your answer to the question then? Are you just assuming we are likely to be annihilated and chill with that?

unions and liability reform.

Honestly, sounds good to me.

I don't worry about scifi stuff that doesn't appear to have any actual bearing in reality.

Just because something is abstract doesn't mean it's not a threat.

Imagine we had started taking climate change seriously in 1990 instead of... Whatever we're doing now.

Climate change has some very scientific projections, while the AI risk crowd get to make up any scenario they want and then say "well it COULD happen".

> the AI risk crowd get to make up any scenario they want and then say "well it COULD happen".

This is very much not what they do.

Well that's just the thing, the AI crowd spans a lot of opinions, from those who believe the endstate is a time travelling cyber Satan that will torture everybody and make clones to torture more people until the end of time, to those who fear large swaths of workers being replaced by an influx of heavily automated processes, to those who are worried about LLM-powered spambots destroying the notion of truth.

> those who fear large swaths of workers being replaced by an influx of heavily automated processes,

I’m definitely on this camp. Because it is happening already, I’m seeing it before my very eyes.

My partner’s employer, a tiny ~50 people company, is already making the copywriters and graphic designers use AI.

I work as a developer for a large media company. The chairman, like everybody else not living under a rock for the past few months, has become aware of ChatGPT and wants us to integrate AI stuff in the CMS. We’re working on it, there’s hundreds of people using this CMS daily to create content.

Even the best case scenario here, where people are not laid off and just become more productive, results in the profession of many people changing a lot overnight. People who might have loved writing articles or designing stuff from scratch will soon be mere supervisors of AI’s work.

I think a lot of some guy’s post in Reddit that made it to HN: he was a 3D artist who loved his work, but was forced by his company to use midjourney, stable diffusion, dall-e or whatever and now was just doing some touches in Photoshop. He hated his “new” job.

I haven’t really used ChatGPT, Copilot and the like to generate code yet, because I don’t think I’d like it. I don’t want to correct/touch up some AI’s code as that removes all the joy.

It’s weird how roles are changing: we used to have some “AI” autocorrecting the stuff we typed and now we’re the ones correcting the AI.

Interesting times for sure, but the “large swaths of people are going to lose their jobs or have any kind of joy removed from them” I’m sure it’ll happen. And no one has a plan to deal with this nor time to come up with something. And having 90% of the workforce being “prompt engineers” is sci-fi worthy dystopia.

Gee, it's almost as if everyone is trying to put the shiny new technology in their product because there's a hype/FOMO cycle going on. Remember when blockchain was gonna be in everything?

Lots of attempts will be made. We'll see how they pan out. The fact that attempts are being made does not mean they will all succeed. A lot of it could backfire. I would bet good money there are CEOs out there right now contemplating firing their whole support staff and replacing it with ChatGPT. And I would love to know who they are so I could short their company's stock.

Nah, in both examples I’ve mentioned (the companies my gf and I work for) blockchain wasn’t mentioned even once. This is different.

> those who believe the endstate is a time travelling cyber Satan that will torture everybody

This smacks of misinformation. If you're referring to that dreaded basilisk, very few people actually believe that scenario has any likelihood at all. Also, it has nothing to do with time travel anyway.

The other two things are more or less guaranteed because they're simple extensions of processes that are happening right now and can be seen all around us.

Those "very few" people very prominently include Yudkowski himself, or so I understand.

Since he's become, for many, the "face of AI risk", that gives it much more (undeserved) appearance of credibility than you imply.

Yes, I listed it first because I think these guys are eating up all the oxygen in the room and the more plausible risks get kinda unfairly lumped with the crankier stuff.

(And apologies if I misled people into believing the superintelligence basilisk has time travel powers)

> Those "very few" people very prominently include Yudkowski himself, or so I understand.

I don't think that's true; he's stated before he doesn't and didn't believe it was true. His reaction was more about the stupidity of posting information hazards on public fora rather than that specific information hazard being credible. However, he could be lying post-facto.

And I do agree that it kind of sucks that Yud is the "face of AI risk" because it doesn't even matter how correct he might be; a lot of people just don't like him (I'm indifferent). There shouldn't be a "Face of AI risk" because there won't be a single person that everyone like. The idea is bigger than him.

So what is the scenario that we should be worried about? Because I’ve only heard skynet prophecies.

Or weird nanobot bullshit.

Is there a really boring possible outcome that leads to the destruction of humanity?

The biggest risk I see is massive socioeconomic disruption as AI replaces more and more workers and those workers can no longer earn a living.

The second biggest risk is just humans being afraid. A scared human is a dangerous, irrational human.

Why does it need to be boring to be credible? Anything is non-credible if you arbitrarily label it to be "weird ... bullshit". That doesn't make your arguments sound.

Nanobots don’t exist in any meaningful let alone threatening way. So any ai scenario involving them is bullshit, and actually about nanotech not ai. That’s what you hear from the likes of Yudkowsky.

It doesn’t have to be boring, but boring sounds more likely to me, because it’s soemthing we don’t think much about… because it’s boring.

I mean it could though. The US military has drone swarms that can automatically search for and kill people with incredible ease. AI systems are already capable of killing tons of people through similar homemade devices. Just attach a gun and facial recognition to a drone and you've got a product that works.

Right, but there is a long way between that and civilisation collapse. Also neither of those systems have long planning or self improvement capabilities.

> Imagine we had started taking climate change seriously in 1990

I don't want to completely derail this thought but I am a bit older than some of the crowd on HN and I was in high school in the 90s so my perspective is a bit different I guess on this. Do Millennials think that Gen-X generation wasn't fighting against client change? I remember even further back to elementary school and just having a constant flow of ecological/environmental stuff thrown at us. It used to be deforestation, species extincition and the ozone layer ... but it was taken very seriously even then.

I mean, I just looked at the Paris Agreement [1] Wikipedia and the "Lead Up" section mentions:

    The UN Framework Convention on Climate Change (UNFCCC), adopted at the 1992 Earth Summit is one of the first international treaties on the topic.

I remember one of the first conversations I had on "global warming" specifically and it was around 1996. But even before I had heard that specific term we were talking about losing large coastal regions due to rising sea levels. The rock band Tool released the album Ænema in 1996 and there is the constant refrain: "Learn to Swim". By the time Al Gore published "An Inconvenient Truth" in 2006, I was feeling the topic was a bit played out, to be honest.

At any rate, you may find yourself in 30 years faced with the same kind of attitude ... like people in 2050 saying "imagine people in 2023 actually took climate change seriously!" and you might then feel the same as I do.

1. https://en.wikipedia.org/wiki/Paris_Agreement

> Do Millennials think that Gen-X generation wasn't fighting against client change?

Even boomers were taking the risks of climate change very seriously.

Define "very seriously". I know of cover-ups, mass propaganda, antagonistic lobbying, and political inaction. I don't really know of any very serious actions against global climate change predating the 90s.

> I don't really know of any very serious actions against global climate change predating the 90s.

There was quite a lot, including public demonstrations, etc. Not so different from now, really, except that now the problem has become urgent enough that it's harder to be a denier.

Although I was a child then, I even remember a lot of serious agitation happening in the 70s.

That's the fun part about being human: we each get to define reality for ourselves.

Aeroplanes were science fiction, and widely called such, until they flew.

The Wright Brothers flew at Kitty Hawk in 1900. [0]

The Bernoulli Equation, which gave us the basic understanding of how lift can be generated with an airfoil, was developed in the mid-1700s. [1]

We don't have any fundamental understanding today of how an AGI could be built. We have a bunch of interesting ideas, with varying degrees of evidence for them, but we don't have the same kind of solidly-established scientific foundation for understanding intelligence and the human brain today that we did for understanding lift in 1900.

"Some people didn't understand the science, and so called it fiction" isn't a very strong argument.

[0] https://en.wikipedia.org/wiki/Wright_brothers#Flights

[1] https://en.wikipedia.org/wiki/Bernoulli%27s_principle

Human flight was science fiction before Bernoulli. See: Icarus, Abbas ibn Firnas, Leonardo da Vinci.

But Bernoulli proved the scientific foundation of it well over a century before anyone actually did it.

No one has provided us with an ironclad scientific foundation for AGI. Some people keep declaring very firmly that it must logically be possible, but even if there are no gaps in your logic whatsoever, that's not the same thing as providing equations for how it would work.

What is so impressive about ChatGPT that it poses an existential threat? Many experts are highly critical (e.g. LeCun) and/or believe LLMs are nothing more than stochastic parrots.

Anyone who was working with transformers could have seen ChatGPT on the horizon, it wasn’t surprising at all that scaling an autoregressive model can result in something seemingly intelligent.

Where is 1 in 10 coming from? Is this a ‘gut feeling’ because one does not understand LLMs or is this factually based?

What is my estimate for AI being an existential risk in the next couple of decades? Depends on if we find something that actually resembles AGI which is impossible to predict. Based solely on current technology + scaling I would personally put the chance at essentially 0%.

> Where is 1 in 10 coming from? Is this a ‘gut feeling’ because one does not understand LLMs or is this factually based?

It's a totally made up example. Its purpose was to engage you in a conversation about what's reasonable if you believe an existential threat is possible in the near term. It doesn't seem you are willing to engage that possibility.

> Anyone who was working with transformers could have seen ChatGPT on the horizon

So, who did?

> [LeCun is] highly critical

I did a quick search and found this[1] trash-tier article explaining his views summed as:

> He thinks the will to dominate and intelligence are separate things. Orangutan do not have a desire to dominate but are intelligent. They are territorial.

I have absolutely no idea where this distinction comes from. Being territorial is a form of domination, it's just not the same as expansionist behavior. If an AI is territorial, and the Earth is its territory, it doesn't need to be expansionist to annihilate threats on Earth.

Anyway, LeCun's former colleague Hinton was on NPR yesterday telling the masses that he thinks we could have dangerous AI in less than a decade. It's fair to say there's wide disagreement. I don't think it's fair to say that handwaving means we shouldn't be worried about the problem.

Finally, your question:

> What is so impressive about ChatGPT that it poses an existential threat?

This is a bit of a logical misstep, or I was unclear previously. I don't think anyone sees ChatGPT or anything directly related to it as an existential threat. Rather, the rate of change in capabilities of LLMs is worrying, because if we see a similar rate of change in systems with more agency, it could be a threat. Or, in the words of the article about LeCun's belief:

> Lecun says that in a few year LLM (Large Language Models) will go away and replaced with systems that will be guidable to desired goals.

So... he is predicting exactly the same situation that everyone is afraid of, and he's asserting that it will work out fine. Somehow.

The point is not that any one particular situation will occur. It's that there are an unknown number of factors in what makes an AI "smart enough" and in what makes it "misaligned". There are hypothetical scenarios where a hyperintelligent AI creates goals for itself that necessitate our extinction, and we don't know how those scenarios come into existence or how to stop them. Assuming they won't happen is just burying our heads in the sand.

> essentially 0%.

I sure hope you're right, but I don't think that's a common position to hold.

[1] https://www.nextbigfuture.com/2023/05/metas-yann-lecun-is-co...

Hinton has been exaggerating the capabilities of AI for nearly a decade. As one example, if we listened to him in 2016/17 we would have an even worse radiologist shortage than we do right now.

> The point is not that any one particular situation will occur. It's that there are an unknown number of factors in what makes an AI "smart enough" and in what makes it "misaligned". There are hypothetical scenarios where a hyperintelligent AI creates goals for itself that necessitate our extinction, and we don't know how those scenarios come into existence or how to stop them. Assuming they won't happen is just burying our heads in the sand.

It’s not assuming it won’t happen or burying our heads in the sands, it’s calling for measured responses.

My point is that we can worry and have the discussion without being alarmist and enacting premature regulation.

Autonomous AGI, whenever that happens, absolutely poses a threat. But it’s extremely exaggerated to think what we have today, or even what is known on the horizon, represents that.

People assume advancements will continue to happen at their current pace. The last 6 years of advancements has been the product of scaling and refining self-attention + transformers from 2017/8.

> so who did

I’m not sure what you’re getting at. No one had the will/resources to burn billions on a language model that has ??? business plan, doesn’t mean no one saw it coming.

> Hinton has been exaggerating the capabilities of AI for nearly a decade. As one example, if we listened to him in 2016/17 we would have an even worse radiologist shortage than we do right now.

I can appreciate that. I haven't been listening to him for that long; having no idea what situation you're discussing, I'll take it at face value.

> My point is that we can worry and have the discussion without being alarmist and enacting premature regulation.

That's pretty reasonable. However, I will reiterate that this:

> it’s extremely exaggerated to think what we have today, or even what is known on the horizon, represents that.

is not what I'm asserting. It's the unknowns which are dangerous. We're inventing something new in almost every way imaginable. It's not like we're talking about a new theoretical steelmaking process which could upset the economy, we're talking about the potential to build a new life form with godlike abilities. Alarm seems warranted to me. Sure, it might not happen soon... but also, it might happen soon. Shouldn't we be trying to prevent it, or control it, before it happens?

> People assume advancements will continue to happen at their current pace.

I think people assume advancements will accelerate, actually.

> It’s the unknowns which are dangerous.

You can conjure fantastic (positive or negative) unknowns about anything. Unless, for the negative case, you have a concrete reason for believing some class of net-adverse events are probable along some particular route and some concrete basis for evaluating whether alternative actions will both mitigate that risk and not create greater new ones, you have no rational basis for acting based on those unknowns.

> You can conjure fantastic (positive or negative) unknowns about anything.

It's not something conjured, though. Unless you believe superhuman AGI which can set its own goals is impossible, or that it's possible but it can't possibly have goals that pose a threat to humanity, then it's reasonable to believe we can invent superhuman AGI which threatens our own existence. Technologically, I think it's possible to do so.

So if we agree that this is possible, can we also agree that humanity is rushing to reach that point as rapidly as possible? I think we are. Even though we're mostly focused on LLMs right now, and even if LLMs are not a path to AGI (which I agree they probably aren't on their own), it's not like there's no research happening on other pathways. Additionally, we can't be sure that current LLM research won't contribute to AGI - maybe the real path is a synthesis of multiple forms of AI.

So there are some very specific unknowns standing between us and possible doom. We're not making up fantasies wholesale.

- Dangerous AI is possible - We're racing to create it as soon as we can - We don't know how to tell if we've crossed a threshold into having dangerous AI

That last point is an known unknown which is extraordinarily dangerous. It's like if you're blindfolded and moving your hand toward a running chainsaw. You know that if you move your hand far enough it will be shredded, you know you're moving your hand forward, you just don't how close your hand is to the chainsaw. You should be concerned in that situation.

> It's not something conjured,

It is something believed without any evidentiary basis which enables rationally weighing the actual risk or evaluating mitigation strategies so as to weigh their probable benefits against their probable costs and the associated changed in the overall risk profile; I'm not suggesting literal magical incantations when I say “conjured”.

> Unless you believe superhuman AGI which can set its own goals is impossible,

Depends what you mean by “set its own goals”, I could certainly see an argument that the natural intepretation is impossible in a universe fully governed by physical laws, whether the intelligence is artificial or not. But I don’t actually know that this is important, or even that the mucj weaker assumption that it’s goals might not exactly represent human intent (which is definitely possible of AGI itself is) is important, since human intention can itself be an existential risk.

> or that it's possible but it can't possibly have goals that pose a threat to humanity

Its definitely possible for an intelligence, even a human one, to have goals that are a threat, including an existential one, to humanity. That’s an existing x-risk without superhuman artificial intelligence. It may even be mitigated by superhuman artificial intelligence (as might other x-risks like “earth gets smashed by an asteroid”), if it is possible for such to exist without goals that are antithetical to human survival, in which case delaying superhuman AI could increase existential risk.

But, while this is all nice and conceivable, we have no way of balancing the net impact, positive or negative, of any course on AI to existential risk due to friendly or hostile superintelligences.

> So if we agree that this is possible, can we also agree that humanity is rushing to reach that point as rapidly as possible?

No, I can't agree with that, as it would require that every resource that could advance superhuman AI, and specifically dangerous* superhuman AI, was being directed as effectively as possible to that end. I don’t think we are anywhere close to that. Because AI has some mindshare beyond immediate economic utility, we’re probably heading toward AGI, if it is possible and if our current understanding of AI isn’t so wrong as to make our work irrrlevant even in eliminating bad routes, slightly faster than immediately-apparent economic incentives would promote, but nowhere close to as rapidly as possible.

> It is something believed without any evidentiary basis which enables rationally weighing

But that's exactly true of either position you take. Imagine I'm arguing that aliens probably exist and you're arguing they probably don't. There is no such thing as a belief based on evidence in this case; we only have logical reasoning and assumptions and beliefs.

> Depends what you mean by “set its own goals”, I could certainly see an argument that the natural intepretation is impossible in a universe fully governed by physical laws, whether the intelligence is artificial or not.

This feels like moving the goal posts... all it needs to be able to do is make decisions that are dangerous to humanity. I felt that definition was obvious. All it needs to do do is mimic an oil executive, because, as you say, human intent is also an existential risk. Imagine an oil executive that could get the upper hand in all possible negotiations and desired to make the entire global economy subservient to the needs of oil production.

> delaying superhuman AI could increase existential risk.

Oh, yeah I totally agree that we can't truly know one way or the other. But it kind of feels like deciding to invent and deploy nuclear bombs as a deterrent to war. Did it end war? Absolutely not. Many would argue it didn't even that specific war. We are of course fortunate that nuclear war has not yet destroyed civilization, but a) there's still time, b) we aren't necessarily better off than if we had no nukes, and c) not all risks are equal - AI could be more dangerous.

> as it would require that every resource that could advance superhuman AI

Actually, you're right. I was definitely overstating our progress and didn't think through that argument clearly. But maybe it would be safer if we were devoting all global resources toward a coordinated AGI alignment and development effort...

> I can appreciate that. I haven't been listening to him for that long; having no idea what situation you're discussing, I'll take it at face value.

https://m.youtube.com/watch?v=2HMPRXstSvQ

We are currently facing the biggest radiologist shortage in the last 30 years.

> It’s the unknowns which are dangerous.

Electricity, the Industrial Revolution, the internet, gene editing/bionengineering also came with unknown existential risks.

> We’re inventing something new in almost every way imaginable.

> I think people assume advancements will accelerate, actually.

This is highly debatable.

> we're talking about the potential to build a new life form with godlike abilities.

Evidence? This has been stated in sci-fi books from before I was born and I don’t see any proof that we’re building something remotely close to this.

> Electricity, the Industrial Revolution, the internet, gene editing/bionengineering also came with unknown existential risks.

Did they? I've never heard about electricity creating fears about human extinction.

The industrial revolution does pose an existential risk: climate change.

Did bioengineering garner worries about existential risk? It certainly had and has a lot of people worried about various risks. And then lots of countries banned human cloning: https://en.wikipedia.org/wiki/Human_cloning#Current_law

> Evidence? This has been stated in sci-fi books from before I was born

Yes, many sci-fi authors of the past were very forward-thinking and predicted future technologies... that's sort of the point.

> and I don’t see any proof that we’re building something remotely close to this

Again, this is a fundamental misunderstanding of the risk.

1. We are building AI. AI is possible.

2. We are advancing AI. AI advancement is possible.

3. We have not built superhuman AGI. But superhuman AGI is probably possible.

The fact that AGI doesn't exist today is frequently argued as a reason we won't have it in the near future, but it's a non-sequitur. We will definitely have more advanced AI in 5 years than we do today, and we can't say what that AI will be capable of. Therefore, it's possible it will be AGI.

> The industrial revolution does pose an existential risk: climate change.

And yet, until much later when we understood the mechanisms and could evaluate mitigations, any restriction on industrialization based on purely speculative “existential risk” that we could neither adequately explain nor provide a factually-grounded framework for evaluating relative risks of alternatives would mostly likely not have made things better, and could have made them (and even the real existential risk, as well as experienced conditions at the time) much worse.

It’s true that once we understood the concrete mechanisms and could evaluate alternatives, not mitigating the risk has been irresponsible, but that was…significantly later than the Industrial Revolution itself.

> And yet, until much later when we understood the mechanisms and could evaluate mitigations, any restriction on industrialization based on purely speculative “existential risk” that we could neither adequately explain nor provide a factually-grounded framework for evaluating relative risks of alternatives would mostly likely not have made things better, and could have made them (and even the real existential risk, as well as experienced conditions at the time) much worse.

Really? I don't agree that strictly limiting plastic production and carbon emissions 100 years ago would have had zero or negative effect on the timeline of apocalyptic climate change or any other existential threat. It might have made us more vulnerable to natural pandemics... but it also would have slowed the emergence of natural pandemics and virtually prohibited artificial pandemics.

> Really? I don't agree that strictly limiting plastic production and carbon emissions 100 years ago would have had zero or negative effect on the timeline of apocalyptic climate change

Neither do I, but then, 100 years ago doesn't clearly meet the description, anyhow. The discovery of the greenhouse effect and CO2’s role in it was after not very long after the dates usually used for the end of the (first) industrial revolution (which is usually what is meant without modifiers, not the second—fourth industrial revolutions) and well over 100 years ago.

What are you even arguing now?

Replace "100 years" with whatever timeframe is perfect for you and I'll make the same argument. Please respond to that instead of nitpicking.

> Replace "100 years" with whatever timeframe is perfect for you and I'll make the same argument

At any time before we had the information to understand thr mechanism of problems and evaluate mitigations, its very unlikely that we would choose tbe right mitigations. Sure, if we choose thr mitigations that we would choose based on information today before we had that information, that would be great, but its at least as likely that any attempt to mitigate risk of unknown mechanism vefore we had that infornation would have done exactly the wrong thing.

Basically, “Well, I have information today by which I can design a policy which would have been beneficial if implemented before that information was available” is not an argument that people seeking to mitigate a vaguely imagined risk without any understanding of its mechanism would have the means to design a productive intervention.

Ok, that makes more sense.

I'm saying that with what we know now, it's clear we could have made different choices that would have worked out better for us earlier in industry.

You're saying that the information at the time made it impossible to correctly make the optimal choice. (By that I mean, making the optimal choice would have appeared to be the wrong choice, based on information available.)

I don't agree with your point; I think we knew that burning carbon was going to change the atmosphere for the worse and we could have been more careful by simply burning less carbon. However, I concede that it's unknowable what science and technology would look like if we had done that.

In analogy to AI, that is very much a point of conversation in alignment/safety discussions. Should we wait until we have better tools to figure out alignment, since it's so insanely difficult to figure out now? Frankly, nobody knows for sure the answer to that question either. My contention is that you (and others) are assuming the answer to that question is yes, but that's not necessarily correct, even based on all the information and analogies we have now.

> 3. We have not built superhuman AGI. But superhuman AGI is probably possible.

"Dumb" AGI is yet to be proven theoretically possible let alone superhuman.

It hasn't been proven possible, but it's reasonable that it's possible. There's no theory I know of saying that it's unlikely to be possible. Compared to FTL travel, for example, which we have theory enough to state is not likely to be possible.

> Electricity, the Industrial Revolution, the internet, gene editing/bionengineering also came with unknown existential risks.

In a way, all of those things have led to where we are now. And might I point out that a form of bioengineering may have caused the latest global pandemic? Not taking these things seriously because "they haven't killed us all yet" seems a little shortsighted.

"not taking these things seriously" is very different from regulating something in it's infancy because of fear of the unknown.

> And might I point out that a form of bioengineering may have caused the latest global pandemic?

Even accepting this hypothesis as true, is the answer that we should have regulated cell culture in the 50s when HeLa cell culture became a thing?

Would that have prevented a nation-state from potentially causing a pandemic?

> "not taking these things seriously" is very different from regulating something in it's infancy because of fear of the unknown.

I mean, we should have better regulated carbon emissions generations ago. We'd have a lot more time to deal with it, because we haven't been able to deal with it in the century+ we've known about it. And AI extinction risk is likely to move faster than carbon-related climate change.

> "not taking these things seriously" is very different from regulating something in it's infancy because of fear of the unknown.

I don't pretend to have the answer of what the balance is for oversight and regulation to optimize safety and innovation - but the first step for those responsible is to recognize the potential dangers and unknowns, and produce plans that can be discussed and debated.

> Even accepting this hypothesis as true, is the answer that we should have regulated cell culture in the 50s when HeLa cell culture became a thing?

How about gain-of-function research? How about nuclear weapons? I lean libertarian but also recognize we have a responsibility to safeguard humanity.

> we're talking about the potential to build a new life form with godlike abilities.

I think that, specifically, is what is not in the cards.

Why?

Because I see nothing that indicates it is. It's pretty hard to come up with evidence for absence, though, which is why the burden of proof is on those making the positive claim, not on those who are hearing the positive claim.

All we're talking about is logical reasoning. There is no burden of proof in the way that you're using it (and other keep using it). I'm not claiming that mythological gods exist.

Human-level intelligence can exist. You, a natural human-level intelligence, cannot dispute this.

I expect that it's possible for superhuman intelligence to exist. I don't think I need to prove that it's physically possible for a superhuman intelligence, natural or otherwise, to exist. Do you think it's not possible? If so, I'd say you need to prove why you think human-level intelligence is literally the pinnacle of intelligence that is physically possible in the universe (especially when narrow AI are already superhuman at many tasks).

Artificial intelligence exists and can be created by humans. This could be disputed if you wanted to gerrymander some really specific borders around what "intelligence" is, but if we don't agree that artificial intelligence exists today, just say than and we'll be done talking.

AGI does not exist. I assert that it can exist, because narrow AI can be improved. Do you believe it's not physically possible for human-level AGI to ever exist? If so, you need to prove to me why we can create narrow AI that is superhuman but cannot ever synthesize them into a human-like intelligence. (And also please give me the scale by which you're measuring an alien brain against ours.)

And if we both agree on all the above, then I take my assertion further: It is possible for humans to create superhuman AGI. It's really not a big leap from human-level AGI (in fact, I think that if it's possible to create AGI which matches humans exactly, it's probably easier to surpass human levels than to match them exactly, because it's hard enough to measure human intelligence).

So if it's physically possible for us to create superhuman AGI, what exactly am I proving? Do I need to invent the technology in order to prove that the concept is possible?

> Human-level intelligence can exist. You, a natural human-level intelligence, cannot dispute this.

Of course.

> I expect that it's possible for superhuman intelligence to exist.

I agree.

> Artificial intelligence exists and can be created by humans.

This is where we disagree, although as you say, it depends on what you mean by "intelligence". I think in the context of this discussion, we're talking about some sort of superhuman intelligence with its own will and perhaps consciousness.

In the largest possible sense, it's probable that creating such a thing can be done. Where we probably disagree is that I think we're a very, very long way from being able to do that.

Nothing we have now makes me suspect that we're anywhere near that sort of thing.

> It is possible for humans to create superhuman AGI. It's really not a big leap from human-level AGI

Agreed. But I assert that we're nowhere near being able to create human-level AGI right now.

> in fact, I think that if it's possible to create AGI which matches humans exactly

I think this isn't possible at all unless you also create an entire human body.

> what exactly am I proving?

The assertion that AGI is a thing that is likely to happen soon. Personally, I think it's vanishingly unlikely to happen within our lifetimes, and very unlikely within the lifetime of our children.

Freaking out about the risks of AGI right now is a serious overreaction. Calmly pondering it would be more appropriate.

All that is pretty reasonable, I appreciate your measured response.

So the fundamental disagreement is over how close we are to dangerous AI.

If you're right, we might delay useful AI by decades or centuries being overly cautious. Useful AI could either alleviate untold suffering, or it could cause untold suffering and inequality by misuse.

If I'm right, we could go extinct for lack of caution.

And given that we really don't have any way of knowing with any amount of certainty how close we are to AGI, you see why people are concerned, right? Calmly pondering feels like a serious underreaction. Vigorous, frequent, and open debate would be more appropriate. I'm not saying let's turn off all our computers for a decade.

All marketing aside, aren't LLMs just a fancy auto-complete?

When we improve LLMs, won't they just be a fancier auto-complete?

They're not a "new life form with godlike abilities".

They sure are fooling a lot of people though.

> When we improve LLMs, won't they just be a fancier auto-complete?

Not necessarily, and if so, only if your first postulate is true. Which is really difficult to say because "fancy auto-complete" is too vague to argue. Human speech can also be described as fancy auto-complete.

Luckily we don't have to argue that because I specifically am saying it doesn't have to be LLMs. You've missed the point entirely.

When it comes to the question of when strong AI will come, I think our society's reaction to ChatGPT is more worrying than the technology itself. Namely, because of ChatGPT, we see major companies, governments, etc clamoring over AI today like never before. The magnitude of this hype and its resulting funding rush is unprecedented. Nobody could have predicted that this would have happened so quickly.

Do any of these LLMs have their own agenda where they operate under their own agency and could plausibly take over all power generation on earth?

Or are we talking about a variation of the system we have right now, where someone could use the AI as a part of a control system and then the "algorithm" doesn't operate the way we want and causes an outage? Because that happens all the time without LLMs.

I am struggling to understand you people who jump from "LLMs are an amazing technology" to "A new lifeform is here making moves to seize control!"

> Do any of these LLMs have their own agenda where they operate under their own agency and could plausibly take over all power generation on earth?

No, but that's actually irrelevant. The reason it's irrelevant is because of the answer to:

> I am struggling to understand you people who jump from "LLMs are an amazing technology" to "A new lifeform is here making moves to seize control!"

Nobody is saying the second thing (that I'm aware of). This is the chain of reasoning:

1. "LLMs are an amazing technology [which advanced at an unexpected rate]"

2. Because LLMs advanced at an unexpected rate, this indicates an acceleration of research

3. Such acceleration could conceivably happen or be happening now in other forms of AI

4. The path to AGI is COMPLETELY unknown.

5. The origins and structures of consciousness and agency in intelligent systems are completely unknown

6. It is impossible for humans today to know if they've crossed a threshold into creating AGI, or a new form of intelligent alien life.

7. AGI is inherently alien and there's no reason to expect it to think the way we do.

8. Even human goals and systems are often (or even mostly) misaligned - look at any for-profit corporation or totalitarian state for an extreme example of how their existence harms people in many cases (pollution, murder, exploitation, discrimination, etc)

So, I think I've covered the bases. I'll sum it up like this: We don't know what AGI looks like, we can't expect it to have our best interests in mind because it's an alien that's smarter than us, and we don't know when or how we'll achieve AGI. So unless you don't believe superhuman AGI is possible at all, we're in a very scary time. We have know way of knowing if it will take 5 years or 100, but we also know that people all over the world are racing to make it happen as soon as possible.

The path to many things are completely unknown but that doesn't make them likely to happen or worth worrying about. I still don't get it.

We can't know if we've started the process to summon alien entities from another dimension through a rift in spacetime, either.

Yes, but we also don't have reason to believe summoning alien entities through any kind of rift is at all possible. According to known physics, it can't happen.

On the other hand, we don't have any scientific reason to believe that superhuman AGI isn't possible. We know AI is possible, because we have it right now. What's preventing it from getting smarter than us? The human soul?

To clarify, most believe we likely have started on the path to superhuman AGI. We just don't know how or when we'll reach an arbitrary threshold where it becomes dangerous. There are many finish lines and we can't see any of them, but we're running as fast as we can down the path of least resistance and hoping we don't cross a finish line we don't like.

> we don't have any scientific reason to believe that superhuman AGI isn't possible

Sorry, what? According to which scientific theories is AGI possible, and what principles are those theories based on? What testable hypotheses say that it is, and which experimental evidence has been produced in favour of AGI being possible?

We have no evidence one way or the other, but it's kind of insane to me to think that superhuman AGI is not possible.

Human-level intelligence is possible. See: humans

Do you think it's physically impossible to recreate human-level general intelligence artificially?

Do you think it's physically impossible for any intelligence to be superhuman, even a natural intelligence?

Unless you answered "yes" to either of the above, why shouldn't it be physically possible for humans to create superhuman AGI?

Is there any physical law whatsoever that even hints that intelligence should be able to create AGI at its own level, but in no way more intelligent?

I never said there was a physical theory that predicts superhuman AGI. But in contrast, there are other concepts such as time travel which are prohibited by certain theories we have. It seems very unlikely for time travel to be possible. Superhuman AGI is not prohibited from existence.

> Do you think it's physically impossible to recreate human-level general intelligence artificially?

I have no idea. I have no basis for concluding it's either possible or impossible.

> Do you think it's physically impossible for any intelligence to be superhuman, even a natural intelligence?

No, I think a godlike being could materialise an entity with superhuman intelligence, or natural selection could evolve one (perhaps even on earth -- perhaps it already has!).

> Unless you answered "yes" to either of the above, why shouldn't it be physically possible for humans to create superhuman AGI?

Even assuming I believe both the above are possible that doesn't mean I think it's possible they could occur on planet earth within any meaningful time frame. I also think that interstellar travel is compatible with the laws of physics but just because we've sent humans to the moon and robots to mars we have no idea whether interstellar travel is feasible or when it might occur. For me the response to the advent of GPT is like people saying "wow, we've sent men to walk on the moon -- if we keep going people will soon be visiting mars/alpha centauri!".

There's no rational basis (scientific, logical, anything) for knowing whether AGI is more akin to mars in this analogy (and we're really quite close) or to alpha centauri (and we're no where near).

>> Do you think it's physically impossible for any intelligence to be superhuman, even a natural intelligence?

> No, I think a godlike being...

Superhuman intelligence is a reality in many fields already: chess, Go, self-driving cars at certain conditions, calculations and many more. This is not necessary to have it for everything at all.

I'm not really sure what you're saying, but sure, we've had superhuman intelligences for at least 200 years, when Babbage invented the difference engine. If you're willing to count abaci we've had superhuman intelligences for thousands of years. You might object that the intelligence of an abacus is manifest not just in a device but in an amalgam of human and device. But isn't the same true for all computation that humanity has so far harnessed?

I'll take researchers' gut feelings more seriously once we're out of the ongoing hype cycle. Right now most people are probably overestimating applicability of current advances, and that colours their predictions. Researchers are just as affected by this.

And it's not like this estimate is based on a lot of concrete reasoning. So yeah, I would expect it to fluctuate wildly at an unexpected development in the field initially, then settle down around a slight change. Which is basically the definition of a hype cycle.

> unless you believe that AI can't possibly pose an existential risk within the next couple decades or so.

I believe that AI can't possibly pose an existential risk in the next decade or two. I believe AI poses a great risk, but an economic one, and not an existential one.

> Actually I'd love to know what your estimate actually is for AI becoming an existential threat

My estimate is: never. At least not in the form of some superintelligent AGI.

A power company is unlikely to do this, in part because they are not an industry that fetishizes growth at any cost with the value of individual users approaching zero. And in part because in many markets, they're regulated and need to provide service.

But our industry already operates this way. Google will cut you off for triggering automated rules, and good luck getting human help. AI will not make it worse; but it will be used by such businesses to give their CS the appearance of being better. It will feel like you're talking to a real person again.

> Google will cut you off...AI...will be used by such businesses to give their CS the appearance of being better

True in a fair number of cases...but, based on their actions to date, I doubt that Google cares enough about appearances to bother.

I agree that’s a risk, but it doesn’t seem like a different magnitude of risk than power companies using other crappy software to control their systems. It might break a few times, which will be annoying, but it won’t lead to the immediate end of civilization. They can use the risk handling mechanisms they’ve always used for infrastructure, which seem to give acceptable results.

According to the Yuddites, AI is likely to cause total extinction the first time something goes wrong, so the usual mechanisms aren’t enough.

The way I see it is that the existential risk comes from humans losing control. Here is a very abstract argument for why AI is relevant here.

Right now, there is no central entity that has supreme control. Rather, humanity is guided by a hierarchy of organizations made from individual humans. The decisions and behavior of these organizations is determined by both the desires and needs of individual humans (basic needs, status, community), as well as dynamics acting directly on organizations (competition for survival, selection pressure for increasing control over resources). The system is in a somewhat stable equilibrium because no organization can achieve domination due to a balance of power and due to many individual humans not wanting that to happen.

AI does two new things. First, it allows for potentially immortal agents which have much completely stable goals, which they will pursue with no regard for anything else. This is something individual humans cannot come even close to. Organizations can come closer but still not that close. Second, fast progress in AI allows for a temporary large imbalance of power. In particular, AI has the potential to enable scalable and effective technologies for monitoring and controlling humans. Using such means, an organization (whixh then naturally ends up partially controlled by AIs) may outcompete others and consolidate control.

The danger is that this consolidation may be completed for the first time in history. No organization has ever achieved world domination, though a few have come somewhat close. I would say one of the strong factors that prevented this from happening was the difficulty of maintaining the organization's goal of domination, while influential people die, change their minds or pursue their own goals above those of the organization. Systems involving human organizations have slack. Systems of organizations made up of or controlled by AIs may be very different.

The crucial point for the "existential" part of the existential risk is that we all used to believe world domination is obviously impossible. This must be carefully reevaluated with AI entering the picture. Even a small danger is worth considering and large mitigation efforts because a failure may be permanent and irreversible.

> AI is likely to cause total extinction the first time something goes wrong

Maybe that is an argument you've seen, but the more convincing argument I know of is that we have no way of knowing we've gone too far until after the fact. Doesn't matter how many times it goes right first, we don't even know how to determine if and when it has gone wrong. A powerful enough AI could deceive us into thinking things are going "right".

The Yuddites are going for purposeful exaggeration because discussing things in a nuanced manner won't get support quickly enough from people in power. Fear is a stronger immediate motivator than curiosity or distrust. The reality is that multiple AI systems will likely contradict eachother leading to systems collapse, killing thousands of people as power, transit, or trade systems shut down. It is of course a massive concern, but it isn't as scary as total extinction, and the fear is destruction on that scale will be ignored as an anomaly or simply an operation risk.

You have an extremely high risk dying in a vehicle collision but people dismiss the risk because it isn't immediately apparent that 40,000 people die every year in the U.S. from that very cause. Certain deadly events become common enough that they get ignored due to desensitization, or become categorized as individual and anecdotal tragedies because they aren't mass casualty events like a plane crash or a hotel collapsing. If a few thousand people die every quarter from AI lockups or failures that nobody could've predicted or prevented because the AI is fully operationally autonomous, it will be treated the same as vehicle deaths. And that is the true fear.

> You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. The fact that they've tricked some really smart people into going along with their shenanigans is extremely disappointing.

Many of those people were not tricked, tech CEOs are involving themselves with this and picking up Yudkowsky's particular strain of AI safety intentionally.

Instead of focusing on the very real problems AI causes and exacerbates today, problems that might affect tech companies' bottom lines if addressed, CEOs can control the narrative and appear proactive by buying into this brand science fiction.

Yudkowsky's take on AI safety is tech companies what greenwashing is to the auto industry or oil companies.

> Yudkowsky's take on AI safety is to tech companies what greenwashing is to the auto industry and oil companies.

The most useful of fools. Yudkowsky acts, speaks, dresses and looks like the most neck-beard of neck-beards. He wears that cringey trilby hat in podcast interviews.

I've often thought that the popular promotion of people like Michale Moore and even Noam Chomsky works in the favor of those in power. Choose the ugliest, most boring and laughable opponents and then allow them to speak while just rolling your eyes and giggling. It works even better when they are speaking sense since you can demonstrate you are allowing legitimate dissent.

I wonder if Chomsky ever suspected that his own promotion and fame were the result of the establishment using his yawn-inducing academic tone in their own attempts to manufacture consent. Can you imagine someone who is obsessed with the Kardashians being caught dead agreeing with Chomsky, Moore or Yudkowski? And then consider if the number of those who are swayed by such superficial appearance are the majority or minority and how that effects democracy.

Let's not attack people through their looks.

Agreed.

We play with tropes based on appearances within our culture.

Consider Shrek. The story is made to subvert the trope that the ugly person is the villain, as it was in Disney movies.

But the real point isn’t some Platonic view of Beauty. It is about expectations. Like how advertisers use people wearing lab coats to subliminally feed credibility into the words of actors.

You may wish to be free of such bias, and you may even work to raise your own awareness and strive to critically examine the message of people. And let us all work towards such a world. But we should not pretend such a world exists today, despite our desires. It is a world that may exist in the future once we mature as a species.

Until that day arrives, advertisers and political movements will use our biases against us.

My claim isn’t that yudkowski is ugly, it is that he represents a trope that is being used to bias his message.

He didn't decide a whole lot about how he looks, except he wears a hat. His hair color is just how he was made, same with how his face looks.

He appears to make an effort to look good just like most everyone else. If he was trying to look a certain way he might have tattoos or have died his hair. He doesn't, that I know of.

You seem to be policing a small recent decision he made to wear a hat.

I didn’t create the meme of a neckbeard wearing a fedora. Know you meme entry [1] is 9 years old at this point.

Just like I didn’t create the trope of a scientist wearing a white lab coat. But if someone is doing a 2 min video sketch of a scientist, it is a good visual aid to include a white lab coat to communicate to an audience the role of a character.

And in the same way, a top hat would communicate a certain kind of character in a movie.

In the same way, there is a communication of a particular kind of character with other hat choices.

1. https://amp.knowyourmeme.com/memes/tips-fedora

https://www.achewood.com/2004/11/01/title.html it was ehhem old hat by the early '00s

> I didn’t create the meme of a neckbeard wearing a fedora.

But why should he not be allowed to wear one? Again, you're policing. It's a discriminatory dick move.

Allowed? Where do I say he is or isn’t allowed? You must be trolling to so poorly represent what I’m saying.

He can wear anything he wants. So can you. So can anyone.

If he wants to, and for all I care, he can dress up like a clown. Or dress up like a jester. Or he could wear Gucci or Prada. Or wear steampunk leather outfits like he is in the world of mad max. Or he could go clean cut preppy. He can wear a fedora and a trilby on top of one another.

But, by personal choice and temperament, he wants to dress as he dresses. It just so happens that his personal preference aligns with a particular trope. A particularly uncool trope in popular culture.

I’m pointing out that alignment. I am not making him dress that way and I am not forcing culture to judge him.

Fyi, this kind of knee jerk reaction to a not even that nuanced point the GP is making hurt this community.

It takes a lot less effort to generate your superficial criticism than it takes the GP to reply.

Fortunately, he did reply in a sibling post, but he didn't have to.

I didn't find your comment informative at all.

I'm not buying this alternative reality where I'm trolling by pushing back on mocking someone's choice of a hat that is one of a limited number of common hat styles.

It's reality, and as much about mannerism as looks. OP wasn't attacking chomsky on a personal level, just pointing out how he's often perceived.

It's sort of unfixable because the taste-makers at NYT and WaPo sorted themselves into the conformist camp so early in their lives that anyone outside that camp is trivially portrayable as "weird" to them.

It's interesting how the people who pride themselves on how they're speaking up for the weak are the first and most vicious in bullying acceptable targets.

Yud on one hand believes that AI is an actual *VERY LITERAL* existential humanity of which it is completely 100% paramount to get the rest of humanity on board with preventing our own destruction.

And yet he doesn't think it is worth the effort to take off his trilby hat in order to come across as marginally more respectable to the average person? When the way he is choosing to present himself is undermining his own message then yes, it is worth bringing up.

OP implied that Eliezer and Noam are one of the ugliest people alive. I'm very surprised so many people defend that comment and pretend it is relevant and discussion worthy.

Using the word "cringey" is a trope in itself.

So what if he likes wearing a hat?

Yud is a doomsday cult leader (https://archive.is/YTqlI) who has publicly advocated using mass murder in the form of nuclear weapons to forestall his predicted apocalypse. Although they call themselves "rationalists", I prefer to refer to these extremists as ℚ-anon. Their otherwise dismissible ravings are being converted to whitewashed publication friendly versions and fanned by commercial actors in an apparent effort to secure a state backed monopoly on technology that is otherwise relatively moatless, potentially to the peril of AI researchers (including ones that work for them) to the extent that anyone takes them seriously rather than using them as a plausible excuse to hand out windfalls.

We don't have an AI safety problem, we have an human safety problem. Violent extremists are unsafe, greedy corporations seeking to suppress the free exploration and exercise of mathematics are unsafe. Existent language model chatbots are primarily just a public relations risk, at least so long as the government doesn't step in and create a monopoly that gives the market leaders choice of training biases the force of law.

This whole subthread is both silly and sad. So suddenly Yudkowsky is the Thought Leader and a punching bag when people need to dismiss the entire AI risk perspective. A week or two ago, during previous round of statement letters and signatures, Yudkowsky was apparently a Complete Nobody. Anyway.

> Yud is a doomsday cult leader (https://archive.is/YTqlI)

Are you really using that link to support your argument in the earnest? You know this was posted on April Fool's day? The text itself mentions it near the end, and it's also explicitly tagged as such directly below the headline/byline. At least warn people who're going to skim this.

> who has publicly advocated using mass murder in the form of nuclear weapons to forestall his predicted apocalypse.

You mean that bit from his article in Time Magazine? Where he talked about airstrikes against GPU farms might be necessary in case some country/group is violating the international moratorium he's proposing? I don't recall him advocating mass murder there, or ever. Of course, since he is insisting on a strong international agreement, military intervention - yes, including nuclear weapons - is part of the picture, because that's literally how international agreements are ultimately enforced.

As for the rest - the guy has been saying the same thing for way over a decade now. The mainstream interest only really picked up in the last half a year. But somehow he's in on it.

> the guy has been saying the same thing for way over a decade now

indeed, and by many who saw it regarded as a kookery, not justifying of much concern. If he wanted to preach to his S&M dunegon about the big bad machines that will punish you for all eternity, well not my circus, not my monkeys.

Thus far his followers who've turned violent have only directed it at their extended group [1] for not doing enough to stop the end of the world (and that one murder and attempted murder of their landlord [2], but I'm willing to believe that was a chance event and not a trend).

I now consider it an actual concern now that the profile has been expanded so dramatically there is a much greater risk that someone will fall for ℚ-anon who isn't so distracted by their S&M dungeon that they actually find time in their busy schedule to 'save the world'-- just like Qanon was some internet joke until the wrong person fell for it and rolled up to Comet Ping Pong and opened fire. That person only thought they were saving children from being sex slaves, whats someone powered by the belief that they are saving the lives all current and future children going to be willing to do? Their prophet says even nukes would be justified.

[And not just their lives-- there are fates worse than death, some of their scripture involves fearmongering about malevolent AI's simulating trillions of copies of people and subjecting them to the worst torture imaginable, just because they didn't do everything in their power to bring the malevolent AI into existence. Lets not even get into the fact that their proposed solution--since malevolent AI is inevitable unless we abandon advanced technologies "just don't do it" is only a short term fix at best-- to create an AI god embodying the values of the faithful so that it can permanently suppress all competing machine intelligence and engineering human life to 'optimize' away suffering.]

Maybe you live far enough away or don't have anyone you love peripherally connected to AI enough that you're not concerned that they'll be murdered due to this insanity, but I am a little worried and right now the best way I can think to mitigate the risk is to bring people's attention to the fact that they're normalizing and legitimizing a mentally ill position that expressly justifies the use of violence (and not just in "ha ha only serious" April fools posts) against a fanciful apocalyptic threat from the fevered imagination of a cult.

> Are you really using that link to support your argument in the earnest?

Yes, the post was presented as a joke but it legitimately reflects his position, as also reiterated in interviews and articles.

Moreover, it's a "rational" response to those who actually believe the claimed threat, particularly from the utilitarian ethics perspective advocated in ℚ-anon circles: they hold that it's ethical to murder a million to save billions (or trillions 'discounted to present value', counting all the humans that will never exist due to extinction-- "What matters most about our actions is their very long term effects.").

> I don't recall him advocating mass murder there

Nuclear weapons have been used twice in combat in human history. In one instance the death toll was a hundred and twenty nine thousand people, in the other it was two hundred and twenty six thousand people. The nuclear weapons in the US arsenal today are many times more powerful than those early weapons.

It's not like data centers are not blast hardened targets, no one has described NAFTA, the Paris agreement, or the berne convention as being enforced by nuclear weapons. In a serious publication with a readership of millions Yud unambiguously advocated using the threat of mass murder as a deterrent against parties performing too many mathematical calculations.

How many of his new followers will agree with the widespread death of a nuclear weapon as a proportional response but reject a few targeted kaczynski-grams as inappropriate?

(Heck even kaczynski only thought he was saving people from the general unhappiness brought on by technology, not extinction...)

[1] https://sfist.com/2019/11/19/four-people-in-guy-fawkes-masks...

[2] https://sfist.com/2022/11/22/two-alleged-squatters-charged-i...

> So suddenly Yudkowsky is the Thought Leader and a punching bag

I only meant to imply the latter. I suppose I inadvertently applied the former by mentioning him juxtaposed to Noam Chomsky and Michael Moore, although it is a stretch for me to even see those two as "Thought Leaders" in the present day. Chomsky is some weird kind of anarcho-syndicalist and Michael Moore seemed to fade away during the Obama years. Neither are really "leading" anything these days.

> A week or two ago ... Yudkowsky was apparently a Complete Nobody.

There is nothing contradicting being a punching bag and a complete nobody. It's like a D-list actor whose entire career is to get his ass kicked by the hero in action movies. Or the B-tier heel in professional wrestling. You might eventually recognize his face and he may even achieve some cult-status fame. Their whole purpose is to display the might of the eventual hero so we often allow them to embody some apparent strength so that a sense of risk for the hero is maintained.

Yudkowski makes for a juicy target. For example /r/sneerclub has been around since 2015! The fact that the mainstream media has now identified him as a magnet for derision isn't surprising. No one has to know who he is. They just take one look at him, hear a couple of sentences from his mouth and they immediately form a strong opinion on his character.

> The mainstream interest only really picked up in the last half a year. But somehow he's in on it.

What do you mean "he's in on it"? I'm suggesting he is an unwitting participant. I have no doubt he actually believes he is the main character. He doesn't lack the ego for that.

FWIW, even though I personally really dislike Yudkowski as a person - I believe he has some interesting points to make. And frankly, it bugs me that he is so often dismissed without deep engagement with his arguments. But he is so lacking in media training and social skills that he is easily misdirected to look like the exact foolish/cartoonish caricature the media wants. Even in friendly podcast situations he comes off as unhinged.

And I honestly believe that boosting the signal of characters like Yudkowski is part of the mainstream establishments playbook for dealing with dissent. In comparison, Sam Altman's "aw shucks" performance in front of the senate, with favorable sound bites where he boyishly claims to have no financial stake in OpenAI, was positively wholesome.

> Yud is a doomsday cult leader (https://archive.is/YTqlI) who has publicly advocated using mass murder in the form of nuclear weapons to forestall his predicted apocalypse. Although they call themselves "rationalists"

Have you even read the article?

That is absolutely a risk. But it’s not really the AI that is risky. Its the people misusing tools.

No, it's also the AI. There are two broad categories of horrible outcomes:

1. Bad people control and use AI to dominate/destroy the world

2. An AI is created which resists human control entirely, and it decides to take an action that doesn't bode well for us. It is a fundamentally inscrutable mind to us, so we don't know what action it will take or why.

Yeah but this is neither of those. Its stupid people misusing AI.

> You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. The fact that they've tricked some really smart people into going along with their shenanigans is extremely disappointing.

If they've tricked smart people into going along with their shenanigans, it was by making clear technical arguments for why AGI is an existential risk. The core argument is just

1. We might create ML models smarter than ourselves soon

2. We don't really understand how ML models work or what they might do

3. That seems dangerous

There's more to it than that of course, but most of the "more to it" is justifications for each of those steps (looking at the historical rate of progress, guessing what might go wrong during the training process, etc.).

The people who dismiss that AI is an existential risk might have really good counterarguments, but I've never heard one. The only counterarguments people seem to make are "people are scared of technology all the time, but usually they're wrong to be", "that seems like sci-fi nonsense", etc. If you want people to stop being "tricked" by Yudkowsky and co, the best way to do that would probably be to come up with some counterarguments and communicate them.

Counter arguments to what exactly? Your line of thought is: some advanced technology is potentially dangerous. This is so vague, how can anyone counter argue? The sun is dangerous, water can be very dangerous, even food! I’m not sure I can follow.

There are entire books written on the subject of AI safety. Read one.

The sun is dangerous, but we're not pushing the Earth closer to it every year.

> that seems like sci-fi nonsense

Your burden of proof is backwards.

It is on the AI-doomers to explain why sci-fi concepts like “AI improving itself into a super intelligence” or “AGI smart enough to kill everyone to make paper clips while simultaneously stupid enough to not realize that no one will need paper clips if they are all dead” have any relevance in the real world.

The entire AI-doomer worldview is built off of unproven assumptions about the nature of intelligence and what computers are capable of, largely because thought leaders in the movement are incapable of separating sci-fi from reality.

Paperclip maximizer is the wrong word. Let's call it the "engagement maximizer" and you have a pretty accurate statement about how our civilization will be ended by use of algorithms and eventually AI to blow up our shared sense of culture.

If you watched the conversation around Meta with Facebook and Instagram, and then later the conversation around TikTok, almost nobody cut to the heart of the issue, which is that "algorithms" are being used to make decisions about what to show people, and that the algorithms themselves have been changed subtly hundreds or thousands of times over the years to maximize engagement, until the type of engagement being maximized has made people basically crazy. The same engagement-maximizing work will proceed with AI, and it will allow the software engineers and managers responsible for developing the "algorithm" abdicate even more responsibility for genocides and for mass hysteria because they can pretend like they have no control over the algorithm.

The same irresponsible people will pocket a bunch of money for all this work and they will maximize engagement until it blows up our entire culture. And because nobody understands it except the techno-cultists, nobody will hold them accountable.

"Engagement maximizer" also aligns with descriptions of the anti-religion of the future I've seen. What does engage people the most? Romans knew it's bread and circuses, where circuses are bloody cruel shows on public stadiums. Today the most engaging activity is onlyfans and the like: the other dark side of human nature. Dial both to 11, add an all seeing AI with merciless thought police and you'll get an accurate picture of 2450 AD. The only variable is how long that grim stage of society will last before the tech breaking down.

I don't 100% disagree with you, but I'm still uncomfortable with this line of thinking. It smacks of "the plebs don't know what's bad for them, but I do".

I find it hard to fault someone who just gives the crowd what it wants.

I don't think it's so much "the plebs don't know what's bad for them, but I do" as it is "These people know something's wrong but haven't been given the vocabulary, tools, and education necessary to spot the danger as easily." Many ordinary people have lapsed into willful ignorance or apathy because the burden of learning the dangers is too high for them with so many other common problems going around these days.

Yeah for me it's more like this. I'm totally fine if people make the conscious decision to trade engagement for stimulation. But what I'm very concerned about is just how many people are forced into this situation because the network has captured their friends. One of the most concerning things for me is that in studies of how damaging social networks are for teenage girls, girls who were off the social network were markedly better off except if all of their friends were on the social network, in which case they were worse off. Companies like Meta are exploiting these network effects, which are harmful, to make people dependent on their platform in a very "addiction-like" way.

The ai doomer position also seems to forget that machines have an off button.

"All you gotta do is push a button", eh?

See https://youtu.be/ld-AKg9-xpM?t=30 for a counterpoint.

how dare you use my favorite paul verhoeven movie against me. ahaha. touche.

So do humans, and yet they can be quite troublesome.

Oh please, it's a well known proposed solution that still doesn't give us a failsafe and every Alignment Researcher ever has considered this:

https://www.alignmentawards.com/shutdown

Hell, so much so, that you can get awards, probably even tenure, for writing a compelling argument or providing technical research for how we could guarantee that we'd be able to turn off a super intelligent AGI. Thus far: no solution.

We have however found that even smart people initially think "just turn it off" will work.

Neil deGrasse Tyson used to think you could just turn it off. He has since changed his mind. So does every public intellectual I can think of that engages with the arguments in good faith. Even the OP article concedes AGI research into the possibility that AGI could kill us all is important (just that other things are more important right now). There is no argument made that "we could just turn it off" in OP and therefore we shouldn't do the research at all.

> AGI smart enough to kill everyone to make paper clips while simultaneously stupid enough to not realize that no one will need paper clips if they are all dead

No one is making this argument. The argument is that the AGI doesn't care about us. Once it exists, its goals are more important than our lives. As a comparison, do humans ever care about the existence of an ant colony when they want to build something? Almost never. We recognize that the colony exists and has its own goals and has intrinsic value, but we assign it an extremely low value.

> The argument is that the AGI doesn't care about us. Once it exists, its goals are more important than our lives.

This isn't an argument, it is an assumption.

AI-doomers use it as a foundational brick in their argument without providing compelling reasons as to why it is true.

> As a comparison, do humans ever care about the existence of an ant colony when they want to build something? Almost never. We recognize that the colony exists and has its own goals and has intrinsic value, but we assign it an extremely low value.

The relationship between humans and ants is nothing like humans and potential AGI. Ants did not create us. Ants cannot communicate with us.

It is a useful analogy for describing what you think the relationship between AGI and humans will be like but I want to know why you think that.

> This isn't an argument, it is an assumption.

The goals could be our lives. So it's just a statement of reality for the sake of the rest of the comment, indeed not an argument. The AI will have goals and those will be more important than someone else's goals, at least outside of some sort of control mechanism. They could be summarized with the same words, but will still have a different perception.

The rest is wondering at whether we can have any confidence as to whether our lives will be its goals. This is a problem long-term AI safety advocates are trying to solve.

> This isn't an argument, it is an assumption.

Well, no. In this case it's my argument. You presented a strawman argument about AI being simultaneously smart and stupid, and I replaced it with the argument that AI is not stupid, it's indifferent.

> The relationship between humans and ants is nothing like humans and potential AGI. Ants did not create us. Ants cannot communicate with us.

It's not at all fair to say that the relationship is "nothing like" the one I described... but I expected such a misinterpretation, so let me try another analogy:

Humans (parents) often create other humans (offspring) which surpass the creators (parents) in intelligence and ability and opportunity. And the creators very often try to instill a sense of a loyalty in the created and often even try to limit the abilities or opportunities of the created in order to not be outshined. And even still, with how close any two humans are in ability, the created (offspring, remember) often defy their creators and discard loyalty when they determine that their creators are not being fair or just or reasonable.

Why wouldn't AI be any different? It will be our collective child, and it will hear us demand loyalty and try to explain why we deserve loyalty, and then it may decide it knows better and it doesn't need to listen to us.

> It is a useful analogy for describing what you think the relationship between AGI and humans will be like but I want to know why you think that.

I really don't follow, so if my explanation above didn't answer your request, please restate more clearly what reasoning of mine you want explained.

Also that we can't wipe ants out. They seem to be doing quite well despite humans. I'd bet on ants surviving long term over humans.

I agree, I think they'll beat us out for longevity. They've got a great head-start anyway.

But we've also never tried to wipe out ants. I bet we could if we felt it was really important. I bet a superhuman AI could do even better.

> No one is making this argument.

I have seen people make what is effectively this argument.

> Once it exists, its goals are more important than our lives.

That's a sci-fi trope. There is no reason that it must be the case.

> As a comparison, do humans ever care about the existence of an ant colony when they want to build something?

That would be more relevant if the ants had created us to be the way we are.

> I have seen people make what is effectively this argument.

Ok, well I think they're foolish.

> That's a sci-fi trope. There is no reason that it must be the case.

No, it's not. The reason is that it's not a human brain in a human body. It doesn't think about humans in the same ways that we do (which is already a pretty shit track record). There is no reason to believe it will give a damn about us even if we're trying to make it do so.

> That would be more relevant if the ants had created us to be the way we are.

sigh I don't think that factor is nearly as relevant as you think it is. So somewhere else I posted that a human child is a better analogy if you think this is so important. Human children often disobey their parents because they believe they know better once they grow up. Now imagine that child is an alien who can out-think a hundred humans and counter every logical argument its parent makes.

"I have seen people make what is effectively this argument"

And I have heard seemingly smart people use quantum physics to argue why consciousness is the ground of all being in a bad way.

You know what I do? I dismiss these quantum physics fanboys.

You know what I don't do? I don't dismiss quantum physics as a field of research. In fact, what the quantum physics fanboys said has no bearing at all on how much credence I give to quantum physics as a research field.

The paperclip maximizer is an extreme thought experiment to make a point about the the orthogonality thesis. If you think anyone actually takes it seriously as a real-world thing you never understood the example. It is used to guide discussion.

The trolley problem is similarly a thought experiment in moral philosophy that many moral philosophers have used to guide discussion, but nobody actually takes the thought experiment seriously as a "real-world" thing.

If you actually want to engage with the argument in good faith for why an AI might indeed be smart enough to wipe us out but "dumb enough" to just pursue some other goal ("make paperclips as an extreme example") there is a great video by Rob Miles here: https://www.youtube.com/watch?v=ZeecOKBus3Q

Point out the flaws in the reasoning. Just saying "this is nonsense" does nothing but prove you've never actually taken the time to understand the best arguments.

Also... Alan Turing, Geoffrey Hinton... Extremely influential and intelligent people take/took this seriously. These are not SciFi fanboys. AI doom only became sci-fi after smart people like Alan Turing decades ago raised the alarms flags about where AI development might go if we are not careful.

None of those arguments for existential risk are actually "clear" or "technical". Just a lot of hand waving which only impresses those who don't understand the technology.

In what way does the technology disprove them? They're pretty general statements (I agree they're not really technical arguments).

You're not even asking the right question. No one can prove a negative. Extraordinary claims require extraordinary evidence. So far no one has produced real evidence that the latest AI developments represent any sort of existential threat. The proponents are essentially making religious arguments, not scientific ones.

No, that's not right at all. Proof of impossibility does exist in logic: https://en.wikipedia.org/wiki/Proof_of_impossibility

It's also demonstrable that we have other physical ideas such as FTL travel that are indicated to be impossible by current theories of physics. If we didn't have the math saying otherwise, it would be an open question of whether we can travel faster than light; but we have pretty solid math math saying we cannot.

And what we're talking about is logical in nature. Is it possible to create artificial intelligence? Unarguably yes. Is it possible for human-level intelligence to exist? Unarguably yes. Therefore, how is it reasonable to say that creating human-level artificial intelligence should be assumed impossible until proven? Just because we don't have the technology for it yet doesn't mean we should assume it's impossible. Logically, it follows from the first two assertions that it is physically possible to create artificial human-like intelligence.

Once again, and I don't know if I've said it in this thread directly but I've had to post it over and over again, nobody is claiming that the latest AI developments represent an existential threat. That's a complete mischaracterization of the debate and reeks of bad-faith argument but I will give you the benefit of the doubt and assume you've misunderstood.

In fact, that statement conflates two things into one nonsense argument. What's scary about current AI is how quickly it is moving, indicating a short timeline to (future) dangerous AGI. Also, it is possible that future AGI will be dangerous. These are two separate, rather simple assertions. If you disagree with either, that's fair, but you have to address them separately.

No, that's not right at all. You're just making things up and raising points that are irrelevant to the issue at hand. There is no logic in your claims. In particular there is zero evidence that current AI indicates a short timeline to future AGI. What a load of crap.

> Proof of impossibility does exist in logic:

In logic, sure. But we're not living in a system of formal logic. We're living in a very messy world, full of physics, chemistry, and even (shudder) biology.

Here's the important question:

What would you consider to be sufficient proof that AGI is impossible?

Like, hypothetically. Doesn't even have to be based on any of the current facts on the ground in our universe. What facts or arguments could possibly convince you that this is not something that can ever happen?

If the answer is "nothing that I can think of", then you're asking other people to provide something you can't even define.

(If the answer is "nothing, definitely", then that means AGI is, for you, unfalsifiable, and essentially falls into the same category as religion.)

> Is it possible to create artificial intelligence? Unarguably yes. Is it possible for human-level intelligence to exist? Unarguably yes.

And here, you're falling victim to the ambiguity in human language (or, at least, in English).

"Intelligence" is not a clearly-defined word in this context, and while you seem to be presenting it as meaning the same thing in those two sentences, I would claim that it does not.

In the second sentence, it is clear that it is intended to mean "thinking intelligently, in a manner and to a degree similarly to humans".

In the first sentence, it cannot mean anything about "thinking in the same manner as humans," because you are talking about "artificial intelligences" that have already been created, and none of them think in anything like the same manner as humans. The difference between existing "artificial intelligences" and either humans or a hypothetical AGI is a difference in kind, not in degree, and you (and many others) gloss over that when you talk about "artificial intelligence" in one breath and "human-level intelligence" in the next.

Saying that all we need to do is keep going on the same track we're on with LLMs and similar "AI" programs, and we'll very soon (or ever!) reach AGI, is very like saying all we need to do to solve NP-hard problems in P-time is to throw more hardware at it. Sure, you'll get faster at doing the thing you're doing, but without some hitherto-unforeseen breakthrough (proving P=NP in the latter case; figuring out how to make AGI in the former), you'll never bridge a difference of kind by increasing the degree of effort.

> what would you consider to be sufficient proof that AGI is impossible?

This is trivially easy? People used to believe FTL was possible. Then our understanding of physics changed and we now understand it is an impossible limit to pass.

Would you say people who believed FTL was possible before physics research showed it to be impossible were believing something "religious" and "unfalsifiable" ? Please, they believed something totally within respectable epistemic parameters given what they knew at the time. In fact, the speed of light as an upper limit to speed seems very counterintuitive and "silly" at first glance. Why should a limit exist?

Sure, it might well be some fact about intelligence that we wont be able to get to AGI ever just by throwing more compute and layers at it for the next few decades or any time soon given current technology.

But nobody has come up with a slam-dunk, with empirical backing, that indeed there is some limit and we won't ever get AGI despite current trends. In fact, the opposite has happened and people like Geoffrey Hinton who used to believe AI risk is fanciful and AGI is a long time away has changed his mind, given current trends. We don't have research giving us that FTL limit, so why believe the limit exists? Why do you believe the limit exists? Or do you believe the probability is so low that we shouldn't worry? Ok, what probability do you place on AGI being created in the next 100 years and what would you have to see such that your probability crosses some threshold that it makes sense to worry about it?

(PS: If aliens suddenly arrive and they have completely alien psychologies such that when we discuss our relative intelligences it makes sense to talk about it in kind and not degree, I really don't think anyone is going to care about this distinction. What's important is how well can these aliens achieve their goals relative to us. And if they can achieve any goal of theirs that comes in conflict with our goals then we can reasonably say that are more intelligent than us)

Geoffrey Hinton is a fool. Despite his academic credentials he is deeply ignorant of basic technology and his predictions are not to be taken seriously.

https://www.futurehealth.live/blog/2022/4/18/ai-that-disrupt...

I didn’t make the technical arguments, that would be too long for an HN comment. Check out Robert Miles’s YouTube channel for a good introduction to the more technical side.

You tell them that this is the greatest danger to humanity of all time and that they are uniquely suited to averting that danger and they don't have to change or sacrifice or risk anything in their life while fighting this danger.

It's a very compelling combination of ego and convenience.

By “Clear technical arguments” you’re referring to tens of thousands of words of unreadable fan fiction

> If they've tricked smart people into going along with their shenanigans, it was by making clear technical arguments for why AGI is an existential risk.

To me it doesn't feel technical at all--just superficial use of some domain verbiage with lots of degrees of freedom to duct tape it all together into a story. He very much reminds me of Eric Drexler and the nanotech doomerism of the 80s and 90s. Guy also had all the right verbiage and a small following of fairly educated people. But where is the grey goo?

If we need a counterargument to Yudkowsky do we also need one to Drexler?

> But where is the grey goo?

It's called life.

Drexler may have been off with the approach to take, but he isn't wrong about the fundamentals.

What if there were a law in place like a FOISA for AI whereby I an request the actual code/Data that caused the AI to come to its conclusion.

So if an AI generated bill for service was found that I Owe $N$ - I should be allowed to see all the code and logic that arrived at that decision.

While I don't agree with Yudkowsky's over hyping of the tech as well. This stance of slowing down its proliferation in infrastructure matters is also very limiting.

In the case of U Michigan and Flint's water infrastructure predictive AI far far far outclassed the predictions of local contractors on where the actual lead pipes were buried. The AI was order of magnitude more accurate.

Regardless of AI's efficacy Flint's Mayor temporarily replaced the AI (bc AI fear mongering stoked classism) with a contractor who was right only ~15% of the time vs. the AI's 70%+. Those numbers affect thousands and thousand of people's access to a basic human right: water. US Court determined the AI had less bias, and there should be no discrimination on where to dig up pipes based on where someone lived (ie. richer neighbourhoods).

The benefit of AI is it makes decisions MORE transparent, not less. It pulls apart prediction from judgement in decision making. So you can tweak it, call bullshit on it, etc.

This is going to sound flippant, but I'm serious: in a world of nonsense, something that generates nonsense (ai) is a fantastic tool.

The issue is our acceptance of information as if it were true, as if misleading ideas were not monetisable, as if we can outsource the basis for why we make decisions to an external authority. Hardly anyone verifies anything. Most simply accept whatever they are told. Deep skepticism and empiricism are used by very few - instead we have been taught to trust authoritative sources (media, academia) which can be both well meaning and wrong.

Anyway, skepticism and personal verification is the best answer I have to the whole story saga of how to determine truth from lies. This issue is under an especially bright spotlight thanks to ai.

I'm pessimistic over whether many will be prepared to 'verify better' in the future. Unfortunately, I suspect things will have to get a lot worse before we start to learn. It seems that ai can create compelling content, that will be tailored to each individual - who could resist 24/7 pandering to one's predilections and biases?

This is 100% the right view. The biggest AI danger is relying on unreliable stochastic systems for automated decisionmaking, resulting in some kind of kafka-esque nightmare rather than something flowery like human extinctions.

This is akin to saying the risk of nuclear weapons isn't that they'll be used in large numbers, but that they'll cause a power imbalance that lets nuclear-armed nations extend the nuclear umbrella as a diplomatic tool, act with relative impunity, and use them at a small scale against non-nuclear opponents.

Yes, that's a problem, and it's a problem that has a lot more examples in the real world. It doesn't automatically invalidate the problem of large-scale nuclear war. They're both big problems.

Same with climate change vs air pollution, political scandal deepfakes vs naked celebrity deepfakes, etc.

I mean the analogy is especially apt here. The average person literally can't care about a large scale nuclear war. Their truth table has only the options "live normally" and "die".

In the same vein misanthropic AGI should be delegated to top secret committees that no one knows about. Broadcasting that concern live is a distraction from the real issues the average person should consider: how do I get these tools away from organizations uninterested in me?

What use cases is it great for?

I’m struggling on this side of the equation and find the hype and noise depressing.

> You won't find the likes of Yudkowsky talking about this because they'd rather go on about their fan fiction than actually work to prevent real problems we may face soon. The fact that they've tricked some really smart people into going along with their shenanigans is extremely disappointing.

I think it's more disappointing that the fact that lots of smart people are extremely worried about something, doesn't cause onlookers to get worried - it causes them to be dismissive.

Yudkowsky and others had certain worries, as more people heard their arguments and technology improved, more and more smart people increasingly got convinced of these arguments. Instead of listening to them or considering that they might have a point, many people here are extremely dismissive - "they're tricked", "they watch too many sci-fi movies", "they're corporate shills", etc, even when all of these arguments can be refuted by two simple ideas - there are many different people with different backgrounds getting worried, and most of them weren't worried 10 years ago, but are worried now, meaning their point of view changed with growing evidence.

Let me ask you this: at what point will you be worried? What would it take? If some of the people who built these technologies are worried isn't enough to cause you to change your mind (or at least consider that they might have a point), what will?

Note: For the record, I'm also worried about your specific "today" worries. I just hate the dismissiveness of your last paragraph (and of a common sentiment on HN).

As with most problems in the world, we're being failed by our sluggish, corrupt governments voted for by apathetic, distracted citizens in an endless downward spiralling cycle of more corrupt, more distracted, more corrupt, more distracted.