> this is such a transparent attention grab (and, by extension, money grab by being overvalued by investors and shareholders)
Ilya believes transformers can be enough to achieve superintelligence (if inefficiently). He is concerned that companies like OpenAI are going to succeed at doing it without investing in safety, and they're going to unleash a demon in the process.
I don't really believe either of those things. I find arguments that autoregressive approaches lack certain critical features [1] to be compelling. But if there's a bunch of investors caught up in the hype machine ready to dump money on your favorite pet concept, and you have a high visibility position in one of the companies at the front of the hype machine, wouldn't you want to accept that money to work relatively unconstrained on that problem?
My little pet idea is open source machines that take in veggies and rice and beans on one side and spit out hot healthy meals on the other side, as a form of mutual aid to offer payment optional meals in cities, like an automated form of the work the Sikhs do [2]. If someone wanted to pay me loads of money to do so, I'd have a lot to say about how revolutionary it is going to be.
[1] https://www.youtube.com/watch?v=1lHFUR-yD6I
[2] https://www.youtube.com/watch?v=qdoJroKUwu0
EDIT: To be clear I’m not saying it’s a fools errand. Current approaches to AI have economic value of some sort. Even if we don’t see AGI any time soon there’s money to be made. Ilya clearly knows a lot about how these systems are built. Seems worth going independent to try his own approach and maybe someone can turn a profit off this work even without AGI. Tho this is not without tradeoffs and reasonable people can disagree on the value of additional investment in this space.
If he crashes it by undermining what is making it hot, he’ll be worth a lot less. Depending on how hard he crashes the party, maybe worthless.
Though this party is going hard enough right now, I doubt he alone could do it.
Ilya is a brilliant researcher who's contributed to many foundational parts of deep learning (including the original AlexNet); I would say I'm somewhat pessimistic based on the "safety" focus — I don't think LLMs are particularly dangerous, nor do they seem likely to be in the near future, so that seems like a distraction — but I'd be surprised if SSI didn't contribute something meaningful nonetheless given the research pedigree.
1. they're so good at showing the appearance of being right;
2. their results are actually quite unpredictable, not always in a funny way;
3. C-level executives actually believe that they work.
Combine this with web APIs or effectors and this is a recipe for disaster.
Just for an example of your danger #1 above. We used to say that the internet always agrees with us, but with Google it was a little harder. ChatGPT can make it so much easier to find agreeing rationalizations.
And the reason I believe that is that, as far as I understand, many companies are laying off employees (or at least freezing hiring) with the expectation that AI will do the work. I have no mean to quantify how many.
It’s like before the end of WWII the world sees the US as a military super power , and THEN we unleash the atomic bomb they didn’t even know about
That is Ilya. He has the tech. Sam had the corruption and the do anything power grab
“Everyone” who works in deep AI tech seems to constantly talk about the dangers. Either they’re aggrandizing themselves and their work, or they’re playing into sci-fi fear for attention or there is something the rest of us aren’t seeing.
I’m personally very skeptical there is any real dangers today. If I’m wrong, I’d love to see evidence. Are foundation models before fine tuning outputting horrific messages about destroying humanity?
To me, the biggest dangers come from a human listening to a hallucination and doing something dangerous, like unsafe food preparation or avoiding medical treatments. This seems distinct from a malicious LLM super intelligence.
But I agree that it’s not at all clear how we get from ChatGPT to the fabled paperclip demon.
The text alone doesn’t do it but add some generated and nearly perfect “spokesperson” that is uniquely crafted to a persons own ideals and values, that then sends you a video message with that marketing .
We will all be brainwashed zombies
This is still "LLMs as a tool for bad people to do bad things" as opposed to "A(G)I is dangerous".
I find it hard to believe that the dangers everyone talks about is simply more propaganda.
That being said, I think we actually agree that AGI doomsday fears seem massively overblown. I just think the current stuff we have is dangerous already.
While this is a new area, it would be naive to write this off as just science fiction.
Of course it is possible that SSI has novel, unpublished ideas.
Think of politics or general social interactions for actual hard mode problems.
The current crop of systems is a product of the transformers architecture - an innovation that accelerated performance significantly. I put the odds another changing everything but I don't think we can entirely discount the possibility. That no one understands these systems cuts both ways.
And that's the key, it is a tool, a tool that will give a lot of power to whoever is controlling it. And that's where safety matters, it should be made so that it helps good guys more than it helps bad guys, and limit accidents. How? I don't know. Maybe people at SSI do. We already know that the 3 laws of robotics won't work, Asimov only made them to write stories about how broken they are :)
Current-gen AIs are already cause for concern. They are shown to be good at bullshitting, something that bad people are already taking advantage of. I don't believe in robot apocalypse, technological singularities, etc... but some degree of control, like we do with weapons is not a bad thing. We are not there yet with AI, but we might be soon.
I also think that we merely got a very well compressed knowledge base, therefore we are far from super intelligence, and so-called safety sounds more Orwellian than having any real value. That said, I think we should take the literal meaning of what Ilya says. His goal is to build a super intelligence. Given that, albeit a lofty goal, SSI has to put safety in place. So, there, safe super intelligence
So out of all potential applications with current-day LLMs, I’m really not sure this is a particularly good one.
Maybe this is fixable if we can train them to cite their sources more consistently, in a way that lets us double check the output?
Why do we need computers that are better than humans at the game of token prediction?
[citation needed]
But that said, there are plenty of crappy, not-AGI technologies that deserve consideration. LLMs can still make for some very effective troll farms. GenAI can make some very convincing deepfakes. Drone swarms, even without AI, represent a new dimension of capabilities for armies, terrorist groups or lone wolves. Bioengineering is bringing custom organisms, prions or infectious agents within reach of individuals.
I wish someone in our slowly-ceasing-to-function US government was keeping a proper eye on these things.
Remove token, and that's what we humans do.
Like, you need to realize that neural networks came to be because someone had the idea to mimic our brains' functionality, and see where that lead to.
Many skeptics at the beginning like you discredited the inventor, but he was proved wrong. LLMs shown how much more than your limited description they can achieve.
We mimicked birds with airplanes, and we can outdo them. It's actually in my view very short sighted to say we can't just mimic brains and outdo them. We're there. ChatGPT is the initial little planes that flew close to the ground and barely stayed up
People don’t ‘think’ the same way, even if some part of how humans think seems to be somewhat similar some of the time.
That is an important distinction.
This is the hype cycle.
I moved about 50% of my capital back into ETFs though before WWDC in case they dumped a turd on the table.
1. They’ve all been tricked/bribed by Sam Altman and company (which btw this is a company started against those specific guys, just for clarity). Including me, of course.
2. You’re not as much of an expert in cognitive science as you think you are, and maybe the scientists know something you don’t.
With love. As much love as possible, in a singular era
1. The "Future of Life" institute is composed of lots of very serious people who recently helped get the EU "AI Act" passed this March, and they discuss the "myriad risks and harms AI presents" and "possibly catastrophic risks". https://newsletter.futureoflife.org/p/fli-newsletter-march-2...
2. Many researchers are leaving large tech companies, voicing concerns about safety and the downplaying of risks in the name of moving fast and beating vaguely-posited competitors. Both big ones like Hinton and many, many smaller ones. I'm a little lazy to scrape the data together, but it's such a wide phenomenon that a quick Google/Kagi should be enough for a vague idea. This is why Anthropic was started, why Altman was fired, why Microsoft gutted their AI safety org, and why Google fired the head of their AI ethics team. We forgot about that one cause it's from before GPT3, but it doesn't get much clearer than this:
> She co-authored a research paper which she says she was asked to retract. The paper had pinpointed flaws in AI language technology, including a system built by Google... Dr Gebru had emailed her management laying out some key conditions for removing her name from the paper, and if they were not met, she would "work on a last date" for her employment. According to Dr Gebru, Google replied: "We respect your decision to leave Google... and we are accepting your resignation."
3. One funny way to see this happening is to go back to seminal papers from the last decade and see where everyone's working now. Spoiler alert: not a lot of the same names left at OpenAI, or Anthropic for that matter! This is the most egregious I've found -- the RLHF paper: see https://arxiv.org/pdf/2203.02155
3. Polling of AI researchers shows a clear and overhelming trend towards AGI timelines being moved up significantly. It's still a question deeply wrapped up in accidental factors like religious belief, philosophical perspective, and general valence as a person, so I think the sudden shift here should tell you a lot. https://research.aimultiple.com/artificial-general-intellige...
The article I just linked actually has a section where they collect caveats, and the first is this Herbert Simon quote from 1965 that clearly didn't age well: "Machines will be capable, within twenty years, of doing any work a man can do.” This is a perfect example of my overall point! He was right. The symbolists were right, are right, will always be right -- they just failed to consider that the connectionists were just as right. The exact thing that stopped his prediction was the frame problem, which is what we've now solved.
Hopefully that makes it a bit clearer why I'm anxious all the time :). The End Is Near, folks... or at least the people telling you that it's definitely not here have capitalist motivations, too. If you count the amount of money lost and received by each "side" in this "debate", I think it's clear the researcher side is down many millions in lost salaries and money spent on thinktank papers and Silicon Valley polycule dorms (it's part of it, don't ask), and the executive side is up... well, everything, so far. Did you know the biggest privately-funded infrastructure project in the history of humanity was announced this year? https://www.datacenterdynamics.com/en/opinions/how-microsoft...
You're of course right that there are more than two options in an absolute sense, I should probably limit the rhetorical flourishes for HN! My argument is that those are the only supportable narratives that answer all known evidence, but it is just an argument.
I don’t have anything against researchers, and I agree I know a lot less about AI than they do. I do however know humans, and not assuming they’re going to take a chance to get filthy rich by doing something so banal is naive.
sorry to disappoint, but human brain fits the same definition
> Your brain does not process information, retrieve knowledge or store memories. In short: your brain is not a computer
> To understand even the basics of how the brain maintains the human intellect, we might need to know not just the current state of all 86 billion neurons and their 100 trillion interconnections, not just the varying strengths with which they are connected, and not just the states of more than 1,000 proteins that exist at each connection point, but how the moment-to-moment activity of the brain contributes to the integrity of the system.
https://aeon.co/essays/your-brain-does-not-process-informati...
Since I don't want to die I am going to say it should be illegal for you to terminate me.
I don't care about an LLM process being terminated so I have no problem with that.
If your response to there not being a big ending cinematic to life with a bearded old man and a church choir, or all your friends (and a penguin) clapping and congratulating you is that you should kill yourself immediately, that’s a you problem. Get in the flesh-golem, shinzo… or Jon Stewart will have to pilot it again.
For those who do believe they are simply fleshy token predictors, is there a moral reason that other (sentient) humans can't kill -9 them like a LLaMa3 process?
And all of the ideas of morality and societal rules come from this desire to survive and desire to survive exists because this is what natural selection obviously selects for.
There is also probably a good explanation why people want to think that they are special and more than prediction engines.
Before you harvest their organs, you might also contemplate whether the very act of questioning one's own sentience might be inherent positive proof.
I'm afraid you must go hungry either way.
It's the opposite; I've always assumed all humans were sentient, since I personally am, but many people in this comment section are eagerly insisting they are, in fact, not sentient and no more than token prediction machines.
Most likely they're just wrong, but I can't peer into their mind to prove it. What if they're right and there's two types of humans, ones who are merely token predictors, and ones who aren't? Now we're getting into fun sci-fi territory.
So much of what we do and say is just pattern fulfillment. Maybe not 100%, on all days.
In short, we believe in free will because we have no choice.
Could current LLM's do any of this?
If you made an LLM to pretend to have a specific personality (e.g. assume you are a religious person and you’re going to make a comment in this thread) rather than “generic catch-all LLM”, they can pretty much do that. Part of Reddit is just automated PR LLMs fighting each other, making comments and suggesting products or viewpoints, deciding on which comment to reply and etc. You just chain bunch of responses together with pre-determined questions like “given this complete thread, do you think it would look organic if we responded with a plug to a product to this comment?”.
It’s also not that hard to generate these type of “personalities”, since you can use a generic one to suggest you a new one that would be different from your other agents.
There are also Discord communities that share tips and tricks for making such automated interactions look more real.
My point was we are not just glorified token predicting machines. There is a lot going on behind what we write and whether we write it or not. Does the method matter vs just the output? I think/hope it does on some level.
Do you remember learning how to read and write?
What are spelling tests?
What if "subjective experience" isn't essential, or is even just a distraction, for a great many important tasks?
My point is just that the evidence for "humans are just token prediction machines and nothing more" is extremely lacking, but there's always someone in these discussions who asserts it like it's obvious.
Does a thermometer predict tokens? It also produces outputs that can be represented as tokens, but it's just a bit of mercury in a tube. You can dissect a thermometer as much as you like and you won't find any token prediction machinery. There's lots of things like that. Zooming out, does that make the entire atmosphere a token prediction engine, since it's producing eg wind and temperatures that could be represented as tokens?
If you need one token per particle then you're admitting that this is task is impossible. Nobody will ever build a computer that can simulate a brain-sized volume of particles to sufficient fidelity. There is a long, long distance from "brains are made of chemicals" to "brains are basically token prediction engines."
To me, the tech has been an amazing breakthrough. The backlash and downplaying by some people seems like some odd type of fear or cope to me.
Even if it is not that world changing, why downplay it like that?
Yep, that’s exactly who I’m talking about! I’m pretty sure Sam Altman is in that camp.
There is a mystery though still - how many people fall for it and then stay fell, and how long that goes on for. People who've followed directly a similar pattern play itself out often many times, and still, they go along.
It's so puzzlingly common amongst very intelligent people in the "tech" space that I've started to wonder if there isn't a link to this ambient belief a lot of people have that tech can "change everything" for the better, in some sense. As in, we've been duped again and again, but then the new exciting thing comes along... and in spite of ourselves, we say: "This time it's really the one!"
Is what we're witnessing simply the unfulfilled promises of techno-optimism crashing against the shores of social reality repeatedly?
Not sure how a Go engine really applies. Do you consider cars superintelligent because they can move faster than any human?
We're just past the Chicago pile days of LLMs [1]. Sutsever believes Altman is running a private Manhattan project in OpenAI. I'd say the evidence for LLMs having superintelligence capability is on shakier theoretical ground today than nuclear weapons were in 1942, but I'm no expert.
Sutsever is an expert. He's also conflicted, both in his opposition to OpenAI (reputationally) and pitching of SSI (financially).
So I'd say there appears to be a disputed but material possibility of LLMs achieving something that, if it doesn't pose a threat to our civilisation per se, does as a novel military element. Given that risk, it makes sense to be cautious. Paradoxically, however, that risk profile calls for strict regulation approaching nationalisation. (Microsoft's not-a-taker takeover of OpenAI perhaps providing an enterprising lawmaker the path through which to do this.)
I can consider the possibility that something coming out of GPU-based neural networks might one day surpass humans in intelligence, but I also believe there's reason to doubt it will be based on today's LLM architecture.
But, all we’ve achieved at this point is making a glorified token predicting machine trained on existing data (made by humans), not really being able to be creative outside of deriving things humans have already made before. Granted, they‘re really good at doing that, but not much else.
To me, this is such a transparent attention grab (and, by extension, money grab by being overvalued by investors and shareholders) by Altman and company, that I’m just baffled people are still going with it.