I think EY's big contribution to the LessWrong/Rationalist cult/movement/ethos is writing the "Sequences," which purport to be a guide to training yourself to think rationally. (https://www.lesswrong.com/tag/original-sequences)

The rationalists are an interesting group. I like a lot of the vibe of the whole thing, but there is also some very weird navel gazing that has cropped up. It can be hard to differentiate from the good and the bad.

I am overall a little surprised by the vitriol towards EY in this thread though. While I agree that the guy is somewhat shrill and his ideas are pretty out there, he really doesn't seem like a con artist from my standpoint. It seems to me that he is genuinely extremely concerned about AI alignment/risk, and his actions seem to make sense in that framework without needing to ascribe a bunch of weird and sinister motivations to him.

Not professing agreement or disagreement with him on any particular thing, just my two cents.

I understand your point, more or less, I just find it funny that people calling themselves "rationalists"/"objectivists"/whatever other funny name just keeps stacking up a bunch of "what if"s and freaking out about it.

As somewhat of a cynical person, I also find it funny that

> While I agree that the guy is somewhat shrill and his ideas are pretty out there, he really doesn't seem like a con artist from my standpoint.

I've heard this exact same sentiment about another popular Effective Altruist in the past year. I will let the reader guess which one.

>"objectivists"/whatever other funny name just keeps stacking up a bunch of "what if"s and freaking out about it.

The Objectivists I know hate hypotheticals and refuge to engage with them with the same dismissive tone used in your comment.

The Objectivists I’ve encountered refuse to engage with lots of things, but still use hypotheticals themselves. They mostly phrase them as generalities with “When…” rather than with “If…”, but they are semantically equivalent.

> I've heard this exact same sentiment about another popular Effective Altruist in the past year. I will let the reader guess which one.

I may be missing something, as I don't really follow this "community" besides reading the occasional Slate Star Codex post or whatever.

But I don't see much ground for comparison here. SBF (who I assume you're referring to) was in much different position with regards to potential harm as a result of con man antics than EY is. (As far as I can tell).

SBF was managing billions of dollars. EY is not. If SBF turns out to be a scammer, real people lose money. If EY is a scammer... then so what? He publishes a bunch of blog posts crying doom about AI that suddenly look less sincere? To what end? I can't see a good reason at this time for assuming his reasons are anything less than what he claims they are.

Eh, I think the reason for his cult-leader status was for creating the whole community and being its, like, prophet. The sequences were just one of many things in that direction.

I'm not surprised by the vitriol, though. He has got to be one of the most arrogant people alive, and people arrogantly writing as though they are world experts on everything when they are, like, dumb and wrong... is extremely irritating.

Systematic use of cult mind control techniques like Scientology and the LaRouche Organization.

Look at all the front groups that are involved in his conspiracy. "Effective Altruism" sounds reasonable at first (donate to causes that can measure their impact) but quickly got hijacked into supporting scam organizations working on scam problems that couldn't possibly measure their output because they are concerned with "longtermist" goals that take as long to observe as watching Plutonium decay.

Charisma, humor, and religious-like ideological framing that can acquire true believers who believe they are saving the world.

He’s an excellent writer and has spent a long time trying to come up with high level easily understandable ways to try to measure alignment abstractly through pure logic/outside of any given system.

Essentially him and his crowd have been trying to create a rationally grounded purely logical non contradictory moral framework for action from axiomatic first principles (he’s not the first or the last to do this), and they believe such a system is essential if AGI comes to be.

I personally think AGI is impossible, his purely logical moral system is impossible, and think what he’s working on is not applicable to current AI (I think he knows this, but his conclusion seems to be that the path to AGI is out of the bag/was made incredibly foolishly and should have been constructed differently with more ability to add guardrails)

But I do think the high level abstract way he thinks is fruitful/very valuable, and a good moderating counterbalance to the frenzied “harder better faster stronger” ML stampede that’s not really thinking about the ramifications of what they’re doing with any sort of discipline.

> I personally think AGI is impossible

That’s a very weird statement to make. Do you think you have a soul or something? Otherwise how is your own Intelligence something a computer can’t replicate?

I don’t think true intelligence is propositional/logic based. I also think the observable universe is inherently limited by our perceptual framework, and there will always be something outside it.

Imagine an amoeba that can only detect light and dark. Its observable universe is a simple gradient. But the cells that are used in its perceptual system cannot be described in a gradient. I think that probably scales up indefinitely.

We confuse what we see for all that exists because of our recent history and scientific advances. The triumph of science comes from focusing solely on what we can see and reason about and improving our vision because thats what we have control over, but it doesn’t mean we see everything. I think true intelligence originates from something we can’t see. You can call it a soul, or call it a scaled up amoeba cell, but I personally think the origin of true intelligence and qualia is from some weird very very old evolutionarily created thing we can’t see (and I think other animals have similar invisible perceptual machinery somewhere, I don’t think it’s just a human thing).

Current AI as exemplified by LLMs like ChatGPT is not propositional/logic based, though, so I'm unsure what your objection is. Artificial intelligence can be just as messy and alogical as natural intelligence. It's not because it runs on a computer that we fully see or understand what it does.

I suppose there is a discussion to be had about what "true intelligence" is and whether some or most human beings have it in the first place.

It’s still propositional and logic based. Every instruction a computer can interpret is propositional and logic based. AI models are chaotic, but deterministic. All computers are deterministic. The distinction I’m making between propositional systems and intelligence is like the distinction between being able to generate a transcendental number like pi (which is propositionally generated/you can calculate it without “understanding” the sequence/being able to reduce it into a smaller pattern) and being able to know whether your procedure to generate pi is correct or incorrect. That latter ability is what I mean by true intelligence, and this transformers approach doesn’t solve that. I believe this is called the “grounding” problem.

Gödel proved it’s impossible for any deterministic system to determine its own correctness close to a hundred years ago. Whatever we’re doing when we say an output A is “right” and output B is “wrong” is fundamentally different than what a propositional deterministic system does, regardless of how large or chaotic or well fitted to expected output it is.

What qualifies as “true intelligence” is the core issue here, yes, and transformers don’t qualify. That doesn’t mean they aren’t valuable or can’t give correct results to things much faster and better than humans, but I think anything we could deliberately design inevitably ends up being deterministic and subject to the introspective limitations of any such system, no matter how big or all encompassing.

I think we’re going to create better and more comprehensive “checkpoint”/“savepoints” for the results of collective intelligence, but that’s different than the intelligence itself.

Our understanding of computer instructions is propositional and logic-based. The hardware itself is a material thing: it's a bunch of atoms and electrons. So is the human brain. Whatever distinction we can make between them cannot be purely conceptual, it has to be physical at some level. The fact that we "logically" designed and made computers, but not brains, is not a relevant distinction. It's not part of the physical reality of either thing, as a physical thing.

> Gödel proved it’s impossible for any deterministic system to determine its own correctness close to a hundred years ago.

I don't think non-deterministic systems fare much better. At the core, completeness is a quantitative problem, not a qualitative one. In order to fully understand and analyze a system of a certain size, there is no way around the simple fact that you need a bigger system. There are more than ten things to know about ten things, so if you only have the ten things to play with, you will never be able to use them to represent a fact that can only be represented using eleven things. Gödel's theorem pushes this to infinity, which is something that we are notoriously poor at conceptualizing. I think this just obfuscates how obvious this limitation is when you only consider finite systems.

Which brings me to the core disagreement that I think we have, which is that you appear to believe that humans can do this, whereas I believe they blatantly cannot. You speak of the grounding problem as if the human brain solves it. It doesn't. Our reasoning is not, has never been, and will never be grounded. We are just pretty damn good at lying to ourselves.

I think that our own capacity for introspection is deeply flawed and that the beliefs we have about ourselves are unreliable. The vast array of contradictory beliefs that are routinely and strongly held by people indicates that the brain reasons paraconsistently: incoherence, contradiction and untruth are not major problems and do not impede the brain's functioning significantly. And so I think we have a bit of a double standard here: why do we care about Gödel and what deterministic systems can or cannot do, in a world where more than half of the population, but let's be honest, it's probably all of us, is rife with unhinged beliefs? Look around us! Where do you see grounded beliefs? Where do you see reliable introspection?

But I'll tell you what. Evolution is mainly concerned with navigating and manipulating the physical world in a reliable and robust way: it is about designing flexible bodies that can heal themselves, immune systems, low-power adaptable neural networks, and so on. And that, strangely enough, is not something AI does well. Why? My theory is that human intelligence and introspection are relatively trivial: they are high impact, for sure, but as far as actual complexity goes, they are orders of magnitude simpler than everything else that we have in common with other animals. Machines will introspect before they can cut carrots, because regardless of what we like to think about ourselves, introspection is a simpler task than cutting carrots.

I think this explains the current trajectory of AI rather well: we are quickly building the top of the pyramid, except that we are laying it directly on the ground. Its failure will not be intelligence, introspection or creativity, it will be mechatronics and the basic understanding of reality that we share with the rest of the animal kingdom. AKA the stuff that is actually complicated.

Yeah, we disagree pretty fundamentally.

Our perception of the physical world is itself an evolved construct. The wider thing that perception is fitting is something I don’t think we can fit a machine to. I think we can only fit it to the construct.

I get where you’re coming from/appreciate the argument, and you may be right, but I‘ve come to appreciate the ubiquity of hidden context more and more/lean that way.

One answer to this would be that humans aren't general intelligences either, but existing in the real world/human society keeps you honest enough you have to try and act like one.

One of the many ways AGI is often defined is “human-level intelligence,” so that seems like a tautological impossibility.

I do think I have a soul, and I don’t think that AGI is impossible.

So, I really don’t get why people think AGI is impossible.

I mean, I have hope that it won’t ever happen, but not because I suspect it is impossible.

I think it’s impossible because I don’t think you can leapfrog evolution, and there’s almost definitely hidden context essential to our intelligence and perception that you can only get through a very very long history.

However I do think it’s quite possible and likely that we’ll create mimics that convince most people they’re intelligent. But I think that’s a weird type of reflection/anything we can make is limited to mirroring our observable past and existing collective thought and can never be truly introspective, because true introspection requires evolutionary context and hidden motivation you can’t transfer.

Evolution is a process that works without an intelligent steward. In a way it's a brute force technique. Plus, nothing is optimizing for intelligence in evolution, it is merely a happy accident that humans ended up with the brains that we did. A different environment could yield drastically different evolutionary history

It doesn't seem very logical to think that because evolution took so long to get us to where we are now, we consequently won't be able to design an intelligent AGI system.

It’d take a while to justify this argument to the extent I think it’s justified, but I think we’re in an inescapable frame set by evolution, and our adaptation to our environment probably goes a lot deeper than what we can see. I don’t think the visible context is the full context, and think true intelligence probably requires an implicit understanding of context which is invisible to us.

A computer could indeed replicate our intelligence, but our intelligence might not be sufficient to write the AGI sourcecode :)

>> Do you think you have a soul or something? Otherwise how is your own Intelligence something a computer can’t replicate?

Strictly speaking, it is for you to say why you think that a digital computer, which is nothing like a human, can be intelligent like a human.

Btw, I see this quip about souls often on HN, in response to comments that AGI is impossible and it is invariably introduced in the conversation by the person arguing that AGI is possible. I have never, ever seen anyone argue that "digital computers can't be intelligent because they don't have a soul". I don't even know where that "soul" bit comes from, probably from something someone said centuries ago? In any case it's irrelevant to most modern conversations where most people don't believe in the supernatural, and have other reasons to think that AGI may be impossible.

And why wouldn't they? For the time being, we don't have anything anyone is prepared to call "AGI". We also don't have a crystal ball to see into the future and know whether it is possible, or not. For all we know, it might be a theoretical possibility, but a practical impossibility, like stable wormholes, or time travel, or the Alcubierre Drive. We will know when we know, not before.

Until then, invoking "souls", or any other canned reply, only seems to serve to shut down curious conversation and should be avoided.

To say it in very plain English: you don't know if AGI is possible, you only assume it to be, so let's hear the other person say what they assume, too. Their opinion is just as interesting as yours, and just as an opinion.

>If scientific knowledge were hidden in ancient vaults (rather than hidden in inconvenient pay-for-access journals), at least then people would try to get into the vaults. They'd be desperate to learn science. Especially when they saw the power that Eighth Level Physicists could wield, and were told that they weren't allowed to know the explanation.

> [...]

>If the Great Secret of Natural Selection, passed down from Darwin Who Is Not Forgotten, was only ever imparted to you after you paid $2000 and went through a ceremony involving torches and robes and masks and sacrificing an ox, then when you were shown the fossils, and shown the optic cable going through the retina under a microscope, and finally told the Truth, you would say "That's the most brilliant thing ever!" and be satisfied. After that, if some other cult tried to tell you it was actually a bearded man in the sky 6000 years ago, you'd laugh like hell.

>And you know, it might actually be more fun to do things that way. Especially if the initiation required you to put together some of the evidence for yourself—together, or with classmates—before you could tell your Science Sensei you were ready to advance to the next level. It wouldn't be efficient, sure, but it would be fun.

>[...]

>And no, I'm not seriously proposing that we try to reverse the last five hundred years of openness and classify all the science secret.

~ Eliezer Yudkowsky, https://www.lesswrong.com/s/6BFkmEgre7uwhDxDR/p/3diLhMELXxM8...

That's really all there is to Yudkowsky's persona of occultism at the end of the day. It's an intentional affectation.

He spends a lot of time writing on LessWrong, noted online agora for the Dunning-Krugers among us.