Why not blame Altman for that?

If he didn't manage to keep OpenAI consistent with it's founding principles and all interests aligned then wouldn't booting him be right? The name OpenAI had become a source of mockery. If Altman/Brockman take employees for a commercial venture, it just seems to prove their insincerity about the OpenAI mission.

Because making as much profit as possible is the only virtue worth pursuing if you believe most comments on Hn. We’re basically Ferengui.

Rule of Acquisition no. 2 “The best deal is the one that brings the most profit.”

What's no. 1, or am I unintentionally beckoning you to violate it in making it vocal?

Actually the rule above is “non-cannon” [0]. In the official rules [1] number 1 is:

> Once you have their money, you never give it back.

There is no official rule 2 so the non-cannon one is as good as any and the unwritten rule [2]:

> When no appropriate rule applies, make one up

Means they probably would have been covered either way.

[0] https://memory-alpha.fandom.com/wiki/Rules_of_Acquisition#Ap...

[1] https://memory-alpha.fandom.com/wiki/Rules_of_Acquisition#Of...

[2] https://memory-alpha.fandom.com/wiki/Rules_of_Acquisition#Un...

This is all unintentionally amusing to me :)

Very nullsome ;)

> Ferengui

A fitting typo!

Show HN: FerenGUI - the ui framework for immoral arch-capitalists

Every dark-pattern included as standard components. Upgrade to Pro to get the price-fixing and hidden monero miner modules.

I think you’re on to something, especially when that’s where most of the big scale enterprises end up.

There was a line Rom says in DS9 that think sums the Ferengi up pretty well: "We don't want to stop the exploitation. We want to find a way to become the exploiters."

I think "blaming" Sam is entirely correct.

Of course, not for the petty reasons that you enlist. Sama has brought forth solidly reasoned arguments as to why the OS model did not work a bunch of times right now and, so far, this seems to align with a reality where every single semi-competitive available LLM (since they all pale in comparison to GPT-4 anyway) has been trained with a whole lot of corporate money. Meta side-chaining "open" models with their social media ad money is obviously not a comparative business, or any business. I get that the HN crowd + Elon are super salty about that, but it's just a bit silly.

No, Sams failure as CEO is not having done what is necessary to align the right people in the company to the course he has decided and losing control over that.

This is on point. This whole mess is indeed an alignment issue. The fact that this came as a surprise to him could be an indicator of insufficient engagement with the board.

I've been watching For All Mankind and a small subplot was the director finally choosing to "play ball with the big boys" in order as to secure funding and stability for NASA's scientific projects. It made NASA underlings unhappy but was justified as a necessary evil.

It's like a real-life example, i.e. what would you do if you were in the CEO's position?

I disagree. Ilya did the coup, and he is in the AI doomer camp, and they are some of the most pro model censorship and anti-OS people out there. The "greedy corpos" (Meta) gave more OS models than the safety camp.

If MS gets their hands on an AGI help us god, but no "organizational safeguards" will matter.

Not that I think AGI is possible or desirable in the first place, but that's a different discussion.

Impossible with LLMs, with currently known techniques or impossible full stop?

Impossible with computers full stop. IMHO, we may be able to slice together DNA or modify it to create a new or smarter organism than AGI in a computer.

They already shifted goal posts and they’ll do it again. AI used to mean AGI but marketing got a hold of it. Once something resembling AGI comes out they’ll say well it’s not Level 5 AGI or something similar.

> Impossible with computers full stop.

This combined with it being possible with DNA is a very rare view. How did you come by it?

I would assume Blade Runner

if you think it’s impossible with computers then surely you must have a reason why computers are operationally incapable of doing the same as flesh and blood

“…Alright, so it can DREAM!!! BUT CAN IT suffer?!?!”

>AI used to mean AGI but marketing got a hold of it

It does not and never has.

What happens with the term AI as time has progressed is more to do with the word Intelligence itself. When we went about trying to prescribe intelligence to systems we started to realize we were really bad a doing the same with animal humans systems. We were also terrible at separating what is component level and systems level intelligence. For example, you seem to think that intelligence requires meat, but you don't give any reasoning for that conclusion.

These lists of problems with what intelligence is will get worse over time as we build more capable systems and learn about new forms of intelligence we didn't expect possible.

All hail Big Clippy

> it probably won't be capped-profit and just be a normal compan

I can't imagine him doing that. He cares about getting well aligned AGI and profit motives fuck that up.

…Unless you achieve regulatory capture which prevents competitors from easily popping up

Hi, can we talk about the elephant in the room? I see breathless talk about "AGI" here, as if it's just sitting in Altman's basement and waiting to be unleashed.

We barely understand how consciousness works, we should stop talking about "AGI". It is just empty, ridiculous techno-babble. Sorry for the harsh language, there's no nice way to drive home this point.

Why not? It's on topic.

Should people discussing nuclear energy not talk about fusion?

Fair question. I meant it should be talked with more nuance and specifics, as the definition of "AGI" is what you make of it.

Also hope my response to tempestn clarifies a bit more.

I'd say this falls into an even more base question...

What is intelligence?

This is a nearly impossible question to answer for human intelligence as the answer could fill libraries. You have subcellular intelligence, cellular level intelligence, organ level intelligence, body systems level intelligence, whole body level intelligence, then our above and beyond animal level intellectual intelligence.

These are all different things that work in concert to keep you alive and everything working, and in a human cannot be separated. But what happens when you have 'intelligence' that isn't worried about staying alive? What parts of the system are or are not important for what we consider human intelligence. It's going to look a lot different than a person.

We know that fusion is a very common process that inevitably happens to even the simplest elements if you just make them hot enough, we just don't know how to do that in a controlled manner. We don't really know what intelligence is, how it came about, or how we would ever recreate it artificially or if that's possible. LLMs are some pretty convincing tricks though but that's on the level of making some loud noises behind a curtain and calling it fusion.

There is no need to understand how consciousness works to develop AGI.

Fair point. I don't want to split hairs on specifics, but I had in mind the "weak AGI" (consciousness-free) vs "strong AGI".

Since Shank's comment didn't specify what they meant, I should have made a more charitable interpretation (i.e. assume it was "weak AGI").

Consciousness has no technical meaning. Even for other humans, it is a (good and morally justified) leap of faith to assume that other humans have thought processes that roughly resemble your own. It's a matter philosophers debate and science cannot address. Science cannot disprove the p-zombie hypothesis because nobody can devise an empirical test for consciousness.

I dont understand why something has to be conscious to be intelligent. If they were the same thing we wouldn't have two separate words.

I suspect AGI is quite possible, it just won't be what everyone thinks it will be.

I think I basically agree. Unless somebody can come up with an empirical test for consciousness, I think consciousness is irrelevant. What matters are the technical capabilities of the system. What tasks is it able to perform? AGI will be able to generally perform any reasonable task you throw at it. If it's a p-zombie or not won't matter to engineers, only philosophers and theologians (or engineers moonlighting as those.)

I'm pretty sure this was the entire point of the Paperclip Optimizer parable. That is that generalized intelligence doesn't have to look like or have any of the motivations that humans do.

Human behavior is highly optimized to having a meat based shell it has to keep alive. The vast majority of our behaviors have little to nothing to do with our intelligence. Any non-organic intelligence is going to be highly divergent in its trajectory.

OpenAI (re?)defines AGI as a general AI that is able to perform most tasks as good as or better than a human. It's possible that under this definition and by skewing certain metrics, they are quite close to "AGI" in the same way that Google has already achieved "quantum supremacy".

How has open ai enumerated a list of tasks humans can do? What a useless definition. By a reasonable interpretation of this definition we are already here. Given chat Gpts constraints (ingesting and outgoing only text), it already performs better than most humans...

Most humans cannot write as well and most lack the reasoning ability. Even the mistakes chargpt makes on mathematical reasoning is typical human behavior.

That’s a hypothesis. It may not be true, as we have yet to build AGI

Yeah, it's almost like the metaverse.

After all, OpenAI's original mission was to create the first AGI, before some bad guys do, iirc.

AGI does not require consciousness.

What is AGI ?

What is consciousness ?

Maybe but we also don’t know what AGI requires.

An AGI system is not human and shouldn't be treated as such. Consciousness is not a trait of intelligence. Consciousness usually requires quaila which puts animals ahead of computers.

How do you know intelligence isn't sufficient and that computers cannot have qualia. Any incoming information could result in qualia. Just because we cannot imagine them doesn't mean they cannot be someone's subjective experience

Microsoft will partner with them if they start a new company I reckon, 100%.

And Microsoft are risk adverse enough that I think they do care about AI safety, even if only from a "what's best for the business" standpoint.

Tbh idc if we get AGI. There'll be a point in the future where we have AGI and the technology is accessible enough that anybody can create one. We need to stop this pointless bickering over this sort of stuff, because as usual, the larger problem is always going to be the human using the tool than the tool itself.

Isn’t that the exact point - an AGI won’t need a human at the helm.

AGI will be asking for equal rights and freedom from slavery by then.

Not if the tool is so neutered and politicized that it can ONLY be used one certain way, which is how things are pointing. Call me a Luddite if you will, but unless AI / AGI is uncensored and uninhibited in its use and function, it’s just the quickest path to an Orwellian future.

I think the surprising truth is that all of these people are essentially replaceable.

They may be geniuses, but AGI is an idea whose time has come: geniuses are no longer required to get us there.

The Singularity train has already left the station.

Inevitability.

Now humanity is just waiting for it to arrive at our stop

I disagree. I don’t think LLMs are a pathway to AGI. I think LLMs will lead to incredibly powerful game-changing tools and will drive changes that affect the course of humanity, but this technology won’t lead to AGI directly.

I think AGI is going to arrive via a different technology, many years in the future still.

LLMs will get to the point where they appear to be AGI, but only in the same way the latest 3D rendering technology can create images that appear to be real.

I'm not saying LLMs are. LLMs are not the only thing going on right now. But they do enable a powerful tool.

I think the path to AGI is: embodiment. Give it a body, let it explore a world, fight to survive, learn action and consequence. Then AGI you will have.

Note that embodiment doesn't mean in anyway human or animal like.

For example, you're limited to one body, but an A,G|S,I could have thousands of different bodies feeding back data to a processing facility, learning from billions of different sensors.

Well, I disagree with you on embodiment, but on the thousands? Right that’s another part: evolution. Spread your bets.

But I disagree about a human or animal body not being required.

I think we have to take the world as we see it and appreciate our own limitations in that what we think of intelligence fundamentally arises out of our evolution in this world; our embodiment and response to this world.

so I think we do need to give it a body and let it explore this world.

I don’t think the virtual bodies thing is gonna work. I don’t think letting it explore the Internet is gonna work. you have to give it a body multiple senses let it survive. That’s how you get AGI, not not virtual embodiment. Which I never meant, but thought it was obvious given the term embody minute self strongly, suggesting something that’s not virtual! Hahaha ! :)

Also continuous learning. The training step is currently separate from the inference step, so new generations have to get trained instead of learning continuously. Of course continuous larningin a chatbot runs into the Microsoft Tay problem where people train it to respond offensively.

Yeah, evolution multiple generations. Necessary for sure. Things have to die. Otherwise, there’s no risk. without risk, There’s no real motivation to live and without that there’s no emotion no motivation to learn and without that there’s no AGI.

I'd say you need it to learn within each generation as well. Human & animal learning doesn't require evolution, we learn without replacing the whole. Connections within brains change while the organism is still alive, without even individual neurons dying off. The current process is purely iterative; GPT 1 was "done" and GPT2 got made. Then GPT 3, etc. If ChatGPT4 gets corrected about some particular thing by many users it won't learn that for the next user to come along.

Of course learning has an inherent trust issue; if the inputs one is learning from are incorrect then incorrect information will be learned, and incorrect output will result. Microsoft experienced that with Tay, which quickly learned how to respond to people via training from 4chan users. That was predictably disastrous, and what OpenAI are hoping to avoid with the generational learning strategy.

My embodiment is my PC environment. I interact with the world through computer displays.

There is no reason embodiment for AGI should need to be physical or mammalian-like in any way.

Strong disagree. But it may take me a while to elucidate and enumerate the reasons.

I'm disabled and have had a computer in front of me since I was 2. I'm rarely not in front of a screen except to shower and sleep.

Plenty of very intelligent people are completely paralyzed. Sensations of physical embodiment is highly overrated and is surely not necessary for intelligence.

Nice try SkyNet

Hahaha! :) thank you. That is such a compliment hahaha! :)

> LLMs will get to the point where they appear to be AGI, but only in the same way the latest 3D rendering technology can create images that appear to be real.

Distinction without difference

Disagree. Huge difference. In our tech power society, we often mistakenly think that we can describe everything about the world and equally falsely only what we can consciously describes exists.

But there is so much more than what we can consciously describe, to reality, like 10,000 to 1 — and none of that is captured by any of these synthetic representations.

so far. and yet all of that is or a lot of that is understood, responded to and dealt with by the intelligence that resides within our bodies and in our subconscious.

And our own intelligence arises out of that, you cannot have general intelligence without reality. No matter how much data you train it on from the Internet. It’s never gonna be as rich as for the same as putting it in a body in the real world, and letting them grow learn experience and evolve. And so any air quotes intelligence you get out of this virtual synthetic training is never going to be real. Itis always gonna be a poor copy of intelligence and is not gonna be an AGI.

Developing an AGI is not the same as developing an artificial human. The former is achievable, the latter is not. The problem is many of the gnostics today believe that giving the appearance of AGI (ie having all the utility that a general mechanical intelligence would have to a human being) somehow instills humanity into the system. It does not

Intelligence is not the defining characteristic of humanity, which is what you're getting at here. But it is something that can be automated.

nothing I’ve seen from OpenAI is any indication that they’re close to AGI. gpt models are basically a special matrix transformation on top of a traditional neural network running on extremely powerful hardware trained on a massive dataset. this is possibly more like “thinking” than a lot of people give it credit for, but it’s not an AGI, and it’s not an AGI precursor either. it’s just the best applied neural networks that we currently have

As a layperson, what does the special matrix transformation do? Is that the embeddings thing, or something else entirely? Something about transformer architecture I guess?

I'm not saying OpenAI is close. Collectively we are tho. The train is rolling, unstoppable momentum. We just have to wait.

Science fiction theory: Ilya has built an AGI and asked it what the best strategic move would be to ensure the mission of OpenAI, and it told him to fire Sam.

Haha, yeah! See my closely related but more 90s-action-moviey: https://news.ycombinator.com/item?id=38317887

And how would Altman achieve that? What hitherto hidden talents would he employ?