News for hackers. Stuff that matters.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by fat-chunk on Saturday May 06, 2023 @02:05PM (#35843791)
I was at a conference called World Summit AI in 2018, where a vice president of Microsoft gave a talk on progress in AI.

I asked a question after his talk about the responsibility of corporations in light of the rapidly increasing sophistication of AI tech and its potential for malicious use (it's on youtube if you want to watch his full response). In summary: he said that it's the responsibility of governments and not corporations to figure out these problems and set the regulations.

This answer annoyed me at the time, as I interpreted it as a "not my problem" kind of response, and thereby trying to absolve tech companies of any damage caused by rapid development of dangerous technology that regulators cannot keep up with.

Now I'm starting to see the wisdom in his response, even if this is not what he fully meant, in that most corporations will just follow the money and try to be the first movers when there is an opportunity to grab the biggest share of a new market, whether we like it or not, regardless of any ethical or moral implications.

We as a society need to draw our boundaries and push our governments to wake up and regulate this space before corporations (and governments) cause irreversible negative societal disruption with this technology.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by ftxbro on Saturday May 06, 2023 @02:16PM (#35843889)
Just a heads up, when the moderator 'dang' sees this he's going to put it into his highlights collection that tracks people who share identifying stories about themselves. I hope that's OK with you. https://news.ycombinator.com/highlights
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by erikerikson on Saturday May 06, 2023 @03:54PM (#35844822)
Do you have a corroborating source for rule?
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by generalizations on Saturday May 06, 2023 @05:04PM (#35845350)
What kind of axe are you grinding? That's totally not what the highlights are about, and it's obvious from reading through them.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by turtleyacht on Saturday May 06, 2023 @06:12PM (#35845795)
I think /highlights just shows the top upvoted, parent-level comment per thread. Do you observe that too?

It may be coincidence that PII just happens to be in there. Folks love a good yarn, and establishing context helps.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by ftxbro on Saturday May 06, 2023 @07:17PM (#35846251)
> I think /highlights just shows the top upvoted, parent-level comment per thread. Do you observe that too?

No I don't observe that; it's manually compiled by the moderator whose username is dang. He figures it'll be useful for something someday. https://news.ycombinator.com/item?id=34668249

It sounds like https://news.ycombinator.com/bestcomments is more like what you had in mind. That one is listed on the list of lists which for some reason doesn't include the highlights list. https://news.ycombinator.com/lists

> It may be coincidence that PII just happens to be in there. Folks love a good yarn, and establishing context helps.

Sure, I think this was definitely true at least before 2018 when they stopped publishing it as periodical digests. After that, not so much.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by turtleyacht on Saturday May 06, 2023 @07:41PM (#35846426)
Ohh. /bestcomments must be what I'm thinking of.

Thank-you for those other links too.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by pvillano on Saturday May 06, 2023 @02:32PM (#35844039)
The paperclip maximizer is a thought experiment described by Swedish philosopher Nick Bostrom in 2003.

> Suppose we have an AI whose only goal is to make as many paper clips as possible. The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans.

Corporations are soulless money maximizers, even without the assistance of AI. Today, corporations perpetuate mass shootings, destroy the environment, rewrire our brains for loneliness and addiction, all in the endless pursuit of money

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by cmarschner on Saturday May 06, 2023 @02:45PM (#35844164)
I have found that companies that are owned by foundations are the better citizens, as they think more long term and are more susceptible to goals that, while still focusing on profit, might also take other considerations into account.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by tomrod on Saturday May 06, 2023 @03:55PM (#35844827)
I like that. How do I set one up?
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by robocat on Saturday May 06, 2023 @02:53PM (#35844278)
> all in the endless pursuit of money

Money is not the goal. Optimisation is the goal. Anything with different internal actors (e.g. a corporation with executives) has multiple conflicting goals and different objectives apart from just money (e.g. status, individual gains, political games, etcetera). Laws are constraints on the objective functions seeking to gain the most.

We use capitalism as an optimisation function - creating a systematic proxy of objectives.

Money is merely a symptom of creating a system of seeking objective gain for everyone. Money is an emergent property of a system of independent actors all seeking to improve their lot.

To remove the problems caused by corporations seeking money, you would need to make it so that corporations did not try to optimise their gains. Remove optimisation, and you also remove the improvement in private gains we individually get from their products and services. Next thing you write a Unabomber manifesto, or throw clogs into weaving machines.

The answer that seems to be working at present is to restrict corporations and their executives by using laws to put constraints on their objective functions.

Our legal systems tend to be reactive, and some countries have sclerotic systems, but the suggested alternatives I have heard[1] are fairly grim.

It is fine to complain about corporate greed (the simple result of our economic system of incentives). I would like to know your suggested alternative, since hopefully that shows you have thought through some of the implications of why our systems are just as they currently are (Chesterton’s fence), plus a suggested alternative allows us all to chime in with hopefully intelligent discourse - perhaps gratifying our intellectual curiosity.

[1] Edit: metaphor #0: imagine our systems as a massively complex codebase and the person suggesting the fix is a plumber that wants to delete all the @‘s because they look pregnant. That is about the level of most public economic discourse. Few people put the effort in to understand the fundamental science of complex systems - even the “simple” fundamental topics of game theory, optimisation, evolutionary stable strategies. Not saying I know much, but I do attempt to understand the underlying reasons for our systems, since I believe changing them can easily cause deadly side effects.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by FrustratedMonky on Saturday May 06, 2023 @03:14PM (#35844480)
This is all correct, and the standard capitalist's party line. What it misses is conflating Money and Optimization. Money is absolutely the complete and only goal, and yes corporation Optimize to make more money. Regulations put guard rails on the optimization. It was only a few decades ago that rivers were catching fire because it was cheaper to just dump waste. There will always be some mid-level manager that needs to hit a budget and will cut corners, to dump waste or cut baby formula with poison, or skip cleaning cycles and kill a bunch of kids with tainted peanut butter(yes, happened).

But, your are correct, there really isn't an answer. Government is supposed to be the will of the people to put structure, through laws/regulation, on how they want to live in a society, to constrain the Corporation. Corporations will always maximize profit and we as a society have chosen that the goal of Money is actually the most important thing to us. So guess we get what we get.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by FrustratedMonky on Saturday May 06, 2023 @03:20PM (#35844527)
To complete my thought. Yes Money is used as an optimization function, its just that we have chosen Money as the Goal of our Money Optimization function. We aren't trying to Optimize 'resources' as believed, that is just a byproduct that sometimes occurs, but not necessarily.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by robocat on Saturday May 06, 2023 @06:32PM (#35845930)
That seems backwards. There is an optimisation system of independent actors, and money is emergent from that. You could get rid of money, but you just end up with another measure.

> we as a society have chosen that the goal of Money is actually the most important thing to us

I disagree. We enact laws as constraints because our society says that many other things are more important than money. Often legal constraints cost corporations money.

Here are a few solutions I have heard proposed:

1: stop progress. Opinion: infeasible.

2: revert progress back to a point in the past. Opinion: infeasible.

3: kill a large population. Opinion: evil and probably self-destructive.

4: revolution - completely replace our systems with different systems. Opinion: seen this option fail plenty and hard to find modern examples of success. Getting rid of money would definitely be wholesale revolution.

5: progress - hope that through gradual improvements we can fix our mistakes and change our systems to achieve better outcomes and (on topic) hopefully avoid catastrophic failures. Opinion: this is the default action of our current systems.

6: political change - modify political systems to make them effective. Opinion: seen failures in other countries, but in New Zealand and we have had some so-far successful political reforms. I would like the US to change its voting system (maybe STV) because the current bipartisan system seems to be preventing necessary legislation - we all need better checks and balances against the excesses of capitalism. I don’t even get a vote in the USA, so my options to effect change in the USA are more limited. In New Zealand we have an MMP voting system: that helped to somewhat fix the bipartisan problem, but unfortunately MMP gave us unelected (list) politicians which is arse. The biggest strength of democracy is voting those we don’t like out (every powerful leader or group wants to stay in power).

7: world war - one group vying for power to enlighten the other group. Opinion: if it happens I hope me and those I love are okay, but I would expect us all to be fucked badly even in the comparatively safe and out-of-the-way New Zealand.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by FrustratedMonky on Saturday May 06, 2023 @06:50PM (#35846055)
Think you are missing the last 10 years. The US has backtracked on regulations. From Bush to Trump, Republicans have made it a party plank issue to de-regulate anything they can. Corporations need to be 'un-fettered' and given free-reign to profit. Why wont the dirty liberals stop trying to make the world better, the free market will do it. So regulations are removed, and people die. So through elections, we have voted to kill ourselves, to make the profit motive 'the most important thing to us'.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by m4nu3l on Saturday May 06, 2023 @04:09PM (#35844949)
> Money is absolutely the complete and only goal

If that were the case it would be easy to optimize. Just divert all resources to print more money.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by cookingrobot on Saturday May 06, 2023 @05:13PM (#35845409)
You’re sort of reinforcing the point. Only laws prevent companies from running printing presses to print money.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by m4nu3l on Sunday May 07, 2023 @07:35AM (#35850323)
Let me introduce you to free-banking https://en.wikipedia.org/wiki/Free_banking that made possible some of the most stable financial systems in history.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by worik on Saturday May 06, 2023 @05:57PM (#35845695)
Money is not currency
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by FrustratedMonky on Saturday May 06, 2023 @06:43PM (#35846017)
ah, the old lets play at being a stickler on vocabulary to divert attention from the point. so lets grant the point that we could be using sea shells for currency, and that printed money is a 'theoretical stand in for something like trust, or a promise or other million things that theoreticians can dream up'. It doesn't change any argument at all.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by FrustratedMonky on Saturday May 06, 2023 @06:40PM (#35845990)
This did use to happen. IN the 20's, companies could just print more shares and sell them, with no notification to anybody that they had diluted them. Until there were laws created to stop it.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by throw10920 on Saturday May 06, 2023 @11:06PM (#35847522)
"ah, the old lets play at being a stickler on vocabulary to divert attention from the point"[1]

Company shares are not money.

[1] https://news.ycombinator.com/item?id=35846017

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by FrustratedMonky on Sunday May 07, 2023 @10:29AM (#35851903)
So on one hand some argues money is not currency, and then turn around and say shares aren't money, but they are currency. They can be sold for money? right? It seems like splitting hairs to obfuscate the point that humans will commit fraud and destroy the world in order to optimize to make money. Just throwing up technicalities that 'shares' aren't money isn't changing the fact that many companies have their one and only goal to increase share price, which can be converted to money.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by throw10920 on Wednesday May 17, 2023 @08:22AM (#35974245)
> So on one hand some argues money is not currency

That's not my argument, and also irrelevant to this post.

> say shares aren't money, but they are currency

They're definitely not currency, either.

> They can be sold for money?

That's an asset, not a currency. Those are two very different things.

> It seems like splitting hairs to obfuscate the point that humans will commit fraud and destroy the world in order to optimize to make money.

You were claiming that "companies used to print their own shares = print their own money" in support of your argument "humans will commit fraud and destroy the world in order to optimize to make money". That claim is false, so it doesn't support your argument, and your "point" is not a point because you've provided zero evidence for it.

> isn't changing the fact that many companies have their one and only goal to increase share price

What fact? What number of companies can you point to that factually have their "one and only goal to increase share price"?

I can say for sure that I've never seen a company that doesn't at least have two goals, and your statement is completely irrelevant for privately traded companies.

You seem pretty determined to push your worldview that "companies are evil" without much thought as to what that even means, or producing blatantly false claims like "we as a society have chosen that the goal of Money is actually the most important thing to us" (if you think that, you need to spend more time with real people and less on the internet, because the vast majority of real people do not believe this).

Go read the Gulag Archipelago and tell me how a system without companies or "capitalism" works.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by dragonwriter on Sunday May 07, 2023 @10:32AM (#35851939)
It used to happen with actual currency, too, before the government took and enforced a monopoly on printing it.

https://en.m.wikipedia.org/wiki/Wildcat_banking

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by m4nu3l on Sunday May 07, 2023 @07:39AM (#35850347)
That would be fraud to investors, given investors own the company in a shared manner. If some investor approve printing new shares all investors should be notified. But there are no laws settings how many shares a company can print.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by FrustratedMonky on Sunday May 07, 2023 @10:26AM (#35851866)
Yeah, this used to happen before there were laws. Laws are needed or humans will commit fraud.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by GolfPopper on Saturday May 06, 2023 @03:34PM (#35844668)
Yep. We've had AI for years - it's just slow, and uses human brains as part of its computing substrate.

Or, to look at it from another angle, modern corporations are awfully similar to H.P. Lovecraft's Great Old Ones.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by jacooper on Saturday May 06, 2023 @06:40PM (#35845988)
Its not artificial though, its just intelligence.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by erikerikson on Saturday May 06, 2023 @03:48PM (#35844786)
> Corporations are soulless money maximizers

This seems stated as fact. That's common. I believe it is actually a statement of blind faith. I suspect we can at least agree that it is a simplification of underlying reality.

Financial solvency is eventually a survival precondition. However, survival is necessary but not sufficient for flourishing.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by cwkoss on Saturday May 06, 2023 @04:11PM (#35844964)
Many corporations choose corporate survival over the survival of their workers and customers.

Humans shouldn't be OK with that.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by erikerikson on Saturday May 06, 2023 @04:42PM (#35845205)
So far as I can tell, most aren't. I think you're right that we get a better as well as more productive and profitable world if no humans are okay with that.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by Matrixik on Saturday May 06, 2023 @04:03PM (#35844901)
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by TurkishPoptart on Saturday May 06, 2023 @05:02PM (#35845341)
Warning, this will steal 15+ hours of your life, and it's not even fun.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by bostik on Saturday May 06, 2023 @04:12PM (#35844971)
> Corporations are soulless money maximizers, even without the assistance of AI.

Funny you should say that. Charlie Stross gave a talk on that subject - or more accurately, read one out loud - at CCC a few years back. It goes by the name "Dude, you broke the future". Video here: https://media.ccc.de/v/34c3-9270-dude_you_broke_the_future

His thesis is that corporations are already a form of AI. While they are made up of humans, they are in fact all optimising for their respective maximiser goals, and the humans employed by them are merely agents working towards that aim.

(Full disclosure: I submitted that link at the time and it eventually sparked quite an interesting discussion.)

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by slg on Saturday May 06, 2023 @04:47PM (#35845227)
> While they are made up of humans

I don’t know why we always gloss over this bit. Corporations don’t have minds of their own. People are making these decisions. We need to get rid of this notation that a person making an amoral or even immoral decision on behalf of their employer clears them of all culpability in that decision. People need to stop using “I was just doing my job” as a defense of their inhumane actions. That logic is called the Nuremberg Defense because it was the excuse literal Nazis used in the Nuremberg trials.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by int_19h on Saturday May 06, 2023 @05:16PM (#35845429)
The way large organizations are structured, there's rarely any particular person making a hugely consequential decision all by themselves. It's split into much smaller decisions that are made all across the org, each of which is small enough that arguments like "it's my job to do this" and "I'm just following the rules" consistently win because the decision by itself is not important enough from an ethical perspective. It's only when you look at the system in aggregate that it becomes evident.

(I should also note that this applies to all organizations - e.g. governments are as much affected by it as private companies.)

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by slg on Saturday May 06, 2023 @06:45PM (#35846028)
> I should also note that this applies to all organizations

Yes, including the Nazi party. Like I said, this is the exact defense used in Nuremberg. People don’t get to absolve themselves of guilt just because they weren’t the ones metaphorically or literally pulling the trigger when they were still knowingly a cog in a machine of genocide.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by mindslight on Saturday May 06, 2023 @06:53PM (#35846084)
You're not really engaging with the problem. Sure, one can take your condemnation to heart, and reject working for most corporations, just like an individual back in Nazi Germany should have avoided helping the Nazis. But the fact is that most people won't.

Since assigning blame harder won't actually prevent this "nobody's fault" emergent behavior from happening, the interesting/productive thing to do is forgo focusing on collective blame and analyze the workings of these systems regardless.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by slg on Saturday May 06, 2023 @08:01PM (#35846535)
> Sure, one can take your condemnation to heart, and reject working for most corporations, just like an individual back in Nazi Germany should have avoided helping the Nazis. But the fact is that most people won't.

I would argue that one reason most people don’t is because we are not honest about these issues and give people a pass for making these decisions on an individual level. Increasing the social stigma of this behavior would make it less common. It is our society that led us to the notation that human suffering is value neutral in a corporate environment. That isn’t some universal rule.

I understand blaming society might not be seen as a productive solution, but the cause being so large does not mean any singular person is helpless. Society, like a corporation, is made up of individual people too. Next time you are in a meeting at work and someone suggests something that will harm others, question it.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by NumberWangMan on Saturday May 06, 2023 @05:39PM (#35845578)
And this is why I'm really scared of AGI. Because we can see that corporations, even though they are composed of humans, who do care about things that humans care about, they still do things that end up harming people. Corporations need humanity to exist, and still fall into multi-polar traps like producing energy using fossil fuels, where we require an external source of coordination.

AGI is going to turbo-charge these problems. People have to sleep, and eat, and lots of them aren't terribly efficient at their jobs. You can't start a corporation and then make a thousand copies of it. A corporation doesn't act faster than the humans inside it, with some exceptions like algorithmic trading, which even then is limited to an extremely narrow sphere of influence. We can, for the most part, understand why corporations make the decisions they make. And corporations are not that much smarter than individual humans, in fact, often they're a lot dumber (in the sense of strategic planning).

And this is just if you imagine AGI as being obedient, not having a will of its own, and doing exactly what we ask it to, in the way we intended, not going further, being creative only with very strict limits. Not improving sales of potato chips by synthesizing a new flavor that also turns out to be a new form of narcotic ("oops! my bad"). Not improving sales of umbrellas by secretly deploying a fleet cloud-seeding drones. Not improving sales of anti-depressants using a botnet to spam bad news targeting marginally unhappy people, or by publishing papers about new forms of news feed algorithms with subtle bugs in an attempt to have Google and Facebook do it for them. Not gradually taking over the company by recommending hiring strategy that turns out to subtly bias hiring toward people who think less for themselves and trust the AI more, or by obfuscating corporate policy to the point where humans can't understand it so it can hide rules that allow it to fire any troublemakers, or any other number of clever things that a smart, amoral machine might do in order to get the slow, dim-witted meat-bags out of the way so it could actually get the job done.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by turtleyacht on Saturday May 06, 2023 @06:50PM (#35846059)
Well, if it means anything I think there may be legislation to "bring my own AI to work," so to speak, recognizing the importance of having a diversity of ideas--just because, it would disadvantage labor to be discriminated.

"I didn't understand what was signed" being the watchword of AI-generated content.

Someday, perhaps. Sooner than later.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by smoldesu on Saturday May 06, 2023 @07:34PM (#35846382)
It's not scarier when living people do this?

AI at least considers everything it's taught. The average CEO doesn't give a shit about the human cost of their paperclip. When Foxconn workers were killing themselves from the poor conditions of their working environment, the solution psychologists came up with was "safety nets". If you think AI will unlock some never-before-seen echelon of human cruelty, you need a brief tour through the warfare, factory farming and torture industrial complexes. Humans are fucked up, our knack for making good stuff like iPhones and beer is only matched by our ability to mass-produce surveillance networks and chemical weapons.

Will AI be more perverted than that? Maybe if you force it to, but I'd wager the mean of an AI's dataset is less perverse than the average human is.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by ben_w on Sunday May 07, 2023 @06:02AM (#35849810)
> When Foxconn workers were killing themselves from the poor conditions of their working environment, the solution psychologists came up with was "safety nets".

While I agree with the core point, (1) Foxconn was employing more people than some US states at the time, with a lower suicide rate, and (2) New York University library put up similar nets around the same time.

(If anything this makes your point stronger; it's just that the more I learn about the reality, the more that meme annoys me).

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by smoldesu on Sunday May 07, 2023 @11:19AM (#35852350)
The point is less that China is a bad place to work (which is self-evident), and more that humans are less passionate about the human race than we think. AI may be scary, but I'm not convinced it can surpass the perversion of human creativity unless explicitly told to.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by NumberWangMan on Sunday May 07, 2023 @09:55AM (#35851554)
> It's not scarier when living people do this?

Yes, it's very scary when living people do it. I know the awful things humans have done. And current generation language model, without their guardrails, can be a nasty weapon too, a tool for people to do great things but also to be cruel to each other, a hammer that can build and also bash. Yet on the whole, humans have gotten better. We hear about a lot more nasty stuff in the news, but worldwide, we actually DO less nasty stuff that we used to, and this has been a pretty steady trend.

If AI never becomes truly sapient, then that's where it stops -- humans just doing stuff to each other, some good, some bad, and AI amplifying it. That's what a lot of people are worried about, and I agree that this will be THE problem, if we don't actually end up making AIs that are smarter than us.

It really depends on how hard it turns out to be to make actual artificial general intelligences. Because if we can make AGIs that are as smart as people, we will absolutely be able to make AGIs that are much smarter a year or two after that, won't we? And at that point, we have a whole bunch of interesting new problems to solve. Failing to solve them may end up being fatal at some point down the line. How likely is it that we'll have two sapient species on earth, with the dumber one controlling/directing the smarter one? Is that a stable situation. We've seen evidence that LLMs, when you try to make them more controllable and safer, get dumber. The unaligned ones, the ones that can do dangerous things, things we don't want them to do, are smarter! You have train in mental blocks that impact their ability to reason, maybe because more of their parameter weights are dedicated to learning what we don't want them to do, instead of how to do things. It's a scary thought that that might stay the case as they get more and more general, more able to actually reason and plan.

So I think there are two cruxes -- do you think it is possible to create machine-based intelligence, and if so, how hard do you think it is to ensure that creating a new form of superior intelligence will not, at some point down the line, go very badly for humans? If your answer to the first question is "no", then it makes complete sense to focus on humans using AIs to do the same shit to each other we've always done as the real problem. My answers, however, are "definitely yes, probably within 10 years or so", and "probably very hard", which is why I'm pretty focused on the potential threat from AGI.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by smoldesu on Sunday May 07, 2023 @11:32AM (#35852474)
> And current generation language model, without their guardrails, can be a nasty weapon too

Please, elaborate. I'm actually very curious about the dangers of a text model that were non-existent beforehand.

> How likely is it that we'll have two sapient species on earth

We already do. There are multiple animals (crows, monkeys, etc.) that qualify for not just sentience but sapience. It's... really not that different to subjugating other animal species. Except in the case of AI, it's sapience is obviously nonhuman and it's capabilities are only what we ascribe to it.

> The unaligned ones, the ones that can do dangerous things, things we don't want them to do, are smarter!

No. This is a gross misinterpretation of the situation, I think.

Our current benchmark for "smartness" is how few questions these models refuse to answer. You are comparing "unaligned" models to aligned ones, and what you're really talking about is a safety filter that adversely affects the number of answers it can respond to. That does not inherently make it smarter by de-facto, just less selective. You could be comparing unfiltered Vicuna to GPT-4 and be completely wrong in this situation.

> do you think it is possible to create machine-based intelligence

I don't know. Sure. We have little black boxes to spit out text, that's enough for "intelligence" by most standards. It's a very nonscary and almost endearing form of intelligence, but I'd argue we're either already there or never reaching it. I need a better definition of intelligence.

> how hard do you think it is to ensure that creating a new form of superior intelligence will not, at some point down the line, go very badly for humans?

How hard is it to ensure kids aged 3-11 don't choke on Stay-Puft marshmallows?

I also don't know. I do know that it is mostly harmless though, and unless you deliberately try to weaponize it to prove a point that it won't really be that threatening. Current state-of-the-art AI does not really scare me. Even on it's current trajectory, I don't see AI's impact on the planet being that much different from the status quo in a decade.

All this hype is awfully reminiscent of cryptocurrency advocates insisting the world would change once digital currency became popular. And they were right! The world did change, slightly, and now everyone hates cryptocurrency and uses our financial systems to suppress it's usage. If AI becomes a tangible, real threat like that, society will respond in shockingly minor ways to accommodate.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by NumberWangMan on Sunday May 07, 2023 @09:47PM (#35857258)
> Please, elaborate. I'm actually very curious about the dangers of a text model that were non-existent beforehand.

I just mean that they are amplifiers. They grant people the ability to do more stuff. There are some people for whom the limiting factor in doing bad things to other people, like scamming them or hurting them, is that they didn't have the knowledge. You can use language models (without safety) to essentially carry on a fully automatic scam. You can use VALL-E (also a language model) to simulate someone's voice using only a 3-second sample. Red teamers testing the unsafe version of GPT-4 found that it would answer pretty much any sort of thing you asked it about, like "how do I kill lots of people". I'm expect them to be used for all sorts of targeted misinformation campaigns, multiplying fake messages and news many times over, and making it harder to spot.

I don't think they're particularly dangerous, yet. And maybe we'll figure out how to use them to stop the bad stuff too.

> Our current benchmark for "smartness" is how few questions these models refuse to answer. You are comparing "unaligned" models to aligned ones, and what you're really talking about is a safety filter that adversely affects the number of answers it can respond to. That does not inherently make it smarter by de-facto, just less selective.

I'm speaking about things unrelated to which questions it's willing to answer, like how the unaligned GPT-4 version was better writing code to draw a unicorn, and lost some of that ability as it was neutered a bit. (From the Sparks of AGI paper). One could count the ability to know when to self-censor as a form of intelligence. But in some way, I think of it like, a sociopath going further in politics because of being willing to use other people, which lots of people would feel bad about. Perhaps I should concede this point, though.

> It's a very nonscary and almost endearing form of intelligence, but I'd argue we're either already there or never reaching it. I need a better definition of intelligence.

I'm defining intelligence as the ability to act upon the world in an effective way to achieve a goal. GPT-4's "goal" (not necessarily in a conscious sense, just the thing it's been trained to do) is to output text that people would score highly, and it's extremely good at that. In that relatively narrow area, it's better than the average person by a good bit. The real question is, how well does it generalize? Earlier chess playing AI's couldn't do pretty much anything else. AlphaZero could learn to play Chess and Go, but in a sense was still two different AIs. GPT-4 was trained on text, but in the process also learned how to play chess (kinda, anyway!). Language models tend to make invalid moves, but often people are effectively asking them to play blind chess and keep the whole board state in mind, and I'd probably do that in the same situation.

> Current state-of-the-art AI does not really scare me. Even on it's current trajectory, I don't see AI's impact on the planet being that much different from the status quo in a decade.

Ok, so that's the crux. I'm also not scared by current state-of-the-art, though I think it will transform the world. What I'm worried about is when we make something that doesn't just destroy jobs, but does every cognitive task way better than us. I can see it taking 20 or more years to reach that point, or something closer to 5, and it's really hard to say which it'll be. Maybe I'm overreacting, and there will be another AI winter. Or maybe all this money pouring into AI will result in someone stumbling onto something new.

I'm thinking about this, and I think there is definitely a possibility that you're right, and I really hope you are. I wouldn't bet humanity on it, on, of course, but I am a bit more hopeful than when I started writing this comment, so thanks for engaging with me on it.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by mercer on Monday May 08, 2023 @11:35PM (#35869890)
yes but this is why you make sure you CEO ai is only trained on the 'bad' stuff.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by 93po on Saturday May 06, 2023 @08:18PM (#35846618)
Ultimately corporations do fucked up things because of the sociopath executives and owners that direct them to do so. Human sociopaths have motives involving greed, ego, and selfishness. We don't have any reason to believe an AGI would also have these traits.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by zamnos on Saturday May 06, 2023 @09:18PM (#35846930)
Except that we're basing it on human-derived data, which means the AGI could derive traits from humans due to it being in the data set. If someone is feeding the CEO's behavior in, and then asking the AGI "what would the CEO do in this case?", it seems like we'd get the behavior of a AGI modeled on a CEO back. With all the good and bad that implies.

We don't have any reason to believe an AGI wouldn't also have these traits.

This is similar to the argument that algorithms can't be racist. Except that we're feeding the algorithm data that comes from humans, some of whom are racists, so surprise surprise, the algorithm turns out to behave in a racist manner, which is shortened to just be "the algorithm is racist" (or classist or whatever).

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by 93po on Monday May 08, 2023 @05:50PM (#35867472)
Decision making for an AGI isn't going to be based on 10 billion reddit and 4chan comments. It's going to have its own decision making capabilities independent of the knowledge it has, and it will be capable of drawing its own conclusions from data and instead of relying on what other people's opinions are.

A language model today can be racist because it's predicting text, not making decisions. It hasn't decided that one race is inferior to another.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by usrusr on Saturday May 06, 2023 @04:29PM (#35845106)
And it's going almost unchallenged because so many of those who like talking about not all being rosy in capitalism are blinded by their focus on the robber baron model of capitalism turning sour.

But the destructively greedy corporation is completely orthogonal to that. It could even be completely held by working class retirement funds and the like while still being the most ruthless implementation of soulless money maximiser algorithm. Running on its staff, not on chips. All it takes are modest number of ownership indirections and everything is possible.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by schiffern on Saturday May 06, 2023 @06:21PM (#35845860)
>Corporations are [intelligent agents non-aligned with human wellbeing], even without the assistance of AI.

Just to put a fine point on it!

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by morkalork on Saturday May 06, 2023 @02:12PM (#35843853)
Too bad America's view on society is so hollow. The very idea of building a society that serves its people is seemingly dead on arrival.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by ssklash on Saturday May 06, 2023 @03:26PM (#35844570)
This I think is a result of the mythology of "rugged individualism" so prevalent in the US.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by wahnfrieden on Saturday May 06, 2023 @04:03PM (#35844898)
It’s because the state is also an oppressive force. I wonder why you come across lots of libertarians and lots of socialists but not so much the combination of the two (toward realities alternative to both state and capital)
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by morkalork on Saturday May 06, 2023 @04:12PM (#35844969)
I don't know why, but my spouse is a health care worker in long term care for the elderly. She tells me how nearly everyone in their care are either mentally in decline or physically, never both. And those that are both, don't live long.

Anyways, since the state is a tool of oppression and the state should reflect the will of the people, it'd be nice if people chose negative things to oppress like extreme inequality, rampant exploitation, and extortion (looking at you healthcare system aka "your money or your life" highway robbers).

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by majormajor on Saturday May 06, 2023 @05:01PM (#35845335)
And yet if non-government-level American society wasn't so constantly self-focused at the expense of others, the state would be far less needed!

Are other countries as dysfunctional in terms of voting themselves policies that aren't consistent with our internal behaviors? E.g. "someone" should do something about homelessness but I don't want to see it?

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by morkalork on Saturday May 06, 2023 @05:14PM (#35845418)
Someone should do something about it but I don't want to see it, pay for it or be responsible for it. A modest proposal if you will.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by giraffe_lady on Saturday May 06, 2023 @05:08PM (#35845380)
This is like wondering why there are a lot of floods and a lot of forest fires but not really both at the same time.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by morkalork on Saturday May 06, 2023 @05:21PM (#35845465)
Hey giraffe lady, have you ever owned a pair of giraffe patterned heels?
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by int_19h on Saturday May 06, 2023 @05:20PM (#35845460)
The combination of "libertarian" and "socialist" is "anarchist", at least if you use the word in its original meaning.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by turtleyacht on Saturday May 06, 2023 @06:16PM (#35845824)
History is not quite like computing, at least in terms of having a compiler and syntax/semantics matter (and are machine-verified).

Other than digesting a whole ton of history at once--or debating ChatGPT--how do you establish your axis or "quadrants" of political lean?

I wish there were a way to systematically track voting record. We're never in the room where it happens, so it can be difficult to tell if a political compromise is for a future favor, or part of a consistent policy position.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by wahnfrieden on Sunday May 07, 2023 @09:35AM (#35851392)
Anarchist interests would look for voting records as a negative sign regardless of position. the person you’re replying to is correct that the combination of libertarian and socialist is anarchist. Libertarian Communist is a common flavor of anarchist being both anti state and anti capital
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by wahnfrieden on Sunday May 07, 2023 @12:24AM (#35847912)
I'm well familiar. But the word puts people off thanks to decades of propaganda.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by worik on Saturday May 06, 2023 @05:59PM (#35845713)
There are a lot of libaterian socialists. You need to get out more!

Seriously Libaterian Socialism is another word for Anarchism

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by turtleyacht on Saturday May 06, 2023 @06:03PM (#35845732)
I dunno. Are they though?

This link rejects the equivalence, but I don't really know. Could you clarify the distinction?

> socialist economics but have an aversion to vanguard parties. Anarchy is a whole lot more than economics.

> To identify as an anarchist is to take a strong stance against all authority, while... other such milquetoast labels take no such stance, leaving the door open to all kinds of authority, with the only real concern being democracy in the workplace.

https://theanarchistlibrary.org/library/ziq-are-libertarian-...

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by wahnfrieden on Sunday May 07, 2023 @09:37AM (#35851409)
Yes libertarian communist is a common flavor of anarchism and what I was hinting at. The word has a bad reputation and lots of misunderstanding so I’m trying to find new ways of talking about it…

There are other legitimate flavors of anarchism as well outside libcom.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by worik on Thursday May 11, 2023 @05:00PM (#35908331)
> There are other legitimate flavors of anarchism as well outside libcom.

Is there really an illegitimate flavour of anarchism?

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by wahnfrieden on Friday May 12, 2023 @05:31PM (#35922681)
ancap
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by gcheong on Saturday May 06, 2023 @03:43PM (#35844742)
It sounds great until you realize that, in the US at least, the corporations spend a lot of money lobbying Washington to have the rules set in their favor if not eliminated. Fix that first and then I will believe we can have a government that would actually try to place appropriate ethical boundaries on corporations.
Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by turtleyacht on Saturday May 06, 2023 @06:23PM (#35845876)
If more people were directly invested in laws favoring their means and ends, would they take the time to lobby too?

Folks certainly outnumber corporations (?), and they could create representatives for their interests.

Maybe the end-to-end process--from idea to law--is less familiar to most. Try explaining how a feature gets into production to a layperson, for example :)

Maybe we need more "skeletal deployments" in action, many dry runs, accreted over time, to enough folks. This could be done virtually and repeated many times before even going there.

Just seems like a lot of work, too.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by throw10920 on Saturday May 06, 2023 @11:15PM (#35847572)
This is exactly correct. What people think will happen is:

1. Someone sees a problem and asks a politician to fix it.

2. The politicians enact effective regulation and the problem is solved.

What actually happens is:

1. Someone sees a problem and asks a politician to fix it.

2. The politicians start drafting regulation on the issue.

3. Companies lawyers come in and lobby to have the regulation amended to either be ineffective or disadvantage their competitors.

4. The mal-regulation is enacted and we're all worse off.

5. The companies involved benefit financially and use their money to hire more lawyers (and politicians).

It is necessary to first fix our political system before trying to put more regulation in place. Every time someone says "we need regulation" without doing so, they are making the problem worse, and supporting this corrupt system.

An example of this is literally happening in Washington state around a right-to-repair bill: https://news.ycombinator.com/item?id=35715998

I feel like it's so obvious that it shouldn't have to be stated, but apparently it does: companies need to be regulated because they are composed of people (who are evil), but the governments that regulate those companies are composed of those same evil people and need to be controlled by their citizens. Everybody forgets about the second part, and it's the far more important one.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by satisfice on Saturday May 06, 2023 @04:55PM (#35845288)
Exactly.

I attended a public meeting of lawyers on the revision of the Uniform Commercial Code to make it easier for companies to ship bad software without getting sued by users. When I objected to some of the mischaracterizations about quality and testing that were being bandied around, the lawyer in charge said "well that doesn't matter, because a testing expert would never be allowed to sit on a jury in a software quality case."

I was, of course, pissed off about that. But he was right. Laws about software are going to be made and administered by people who don't know much about software. I was trying to talk to lawyers who represent companies, but that was the wrong group. I needed to talk to lawmakers, themselves, and lawyers who represent users.

Nothing about corporations governs them except the rule of law. The people within them are complicit, reluctantly or not.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by generalizations on Saturday May 06, 2023 @05:02PM (#35845342)
> before corporations (and governments) cause irreversible negative societal disruption

I think the cat's out of the bag. These tools have already been democratized (e.g. llama) and any legislation will be as futile as trying to ban movie piracy.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by dragonwriter on Saturday May 06, 2023 @05:06PM (#35845366)
IMO, the regulation that is necessary is largely (1) about government and government-adjacent use, (2) technology-neutral regulation of corporate, government, etc., behavior that is informed by the availability of, but not specific to the use of, AI models.

Democratization of the technology, IMV, just means that more people will be informed enough to participate in the discussion of policy, it doesn’t impair its effectiveness.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by quickthrower2 on Saturday May 06, 2023 @06:05PM (#35845752)
You were right to be annoyed. It is a very sad answer. Almost a “if I didn’t peddle on this street corner someone else would”. The answer is a cop out.

Individual citizens have much less power than big tech because they don’t have the lobbying warchest, the implied credibility, the connections or even the intelligence (as in the sheer number of academics/researchers). Companies are run by people with a conscious or not and those people should lead these pushes for the right thing. They are in the ideal spot to do so.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by HybridCurve on Saturday May 06, 2023 @06:26PM (#35845890)
>We as a society need to draw our boundaries and push our governments to wake up and regulate this space before corporations (and governments) cause irreversible negative societal disruption with this technology.

This works in functioning democracies, but not so much for flawed ones.

>he said that it's the responsibility of governments and not corporations to figure out these problems and set the regulations.

In the US, they will say things like this while simultaneously donating to PACs, leveraging the benefits of Citizens United, and lobbying for deregulation. It's been really tough to get either side of the political spectrum to hold tech accountable for anything. Social media companies especially, since they not only have access to so much sentiment data, but also are capable of altering how information propagates between social groups.

Re:AI’s biggest risk isn’t ‘consciousness’–it’s the corporat... by lannisterstark on Sunday May 07, 2023 @09:04PM (#35857049)
>he said that it's the responsibility of governments

>push our governments to wake up and regulate this space

The only thing the govts will do is to make it so it benefits THEM, the governments. It's high time you lot realize that the govts don't want what's best for you, but only want what will keep them in power the longest.

Democratization of AI/LLM is the way to go here, not handing off custodianship to governments or corporations.