He's clearly a bright guy, but a lot of his work seems to be trying to reconcile Old Testament narrative patterns with atheism, and simply slotting an omnipotent AI in as a replacement.
Having been raised substantially by a Calvinist grandmother, oh yeah.
It's a super weird religion.
- everyone is damned and depraved
- god chose some people to save, at his own whim, for no reasons related to their actions
- jesus' death was a sacrifice to atone for those people, not for everyone
- the chosen can't opt-out (I'm merging two into one here)
I don't really see the connection to effective altruism at all. The way I grew up understanding it is actually more the opposite?
When EA met Silicon Valley the basics of utilitarianism, minimize suffering, merged with the standard exponential curve that Silicon Valley loves. It went from mosquito nets prevent malaria for pennies to the future is infinite therefore there will always be more people in the future than the present so minimizing future peoples suffering is key. This is where it starts to combine with super intelligence. If there is a super intelligence that destroys humanity that would cause _maximum suffering_. Therefore the most effective way to reduce suffering is actually to prevent said super intelligence from destroying/torturing humanity. Again because there’s more future people than current people.
This is how it becomes Calvinist. The present has no value. Only the chosen, those working in AI, can save the future. And you must do everything you can to prevent said future.
It can go both ways. Either you believe ASI will always be bad so you believe work must be done to prevent it. Or you believe that ASI can be made good and the good kind will maximize utility. They are both the same though fundamentally and are a belief in a higher power, ASI, which either provides ultimate damnnation and must be averted or provides ultimate salvation and must be brought into existence.
If you’re donating mosquito nets or reading up on the best charities in your area that’s not Calvinist. If you’re devoting your life to AI because you believe that’s how you can be most altruistic it starts to look something like a religion and Calvinism is an imperfect but useful analog.
Brian Johnson is also an interesting case of this with his longevity / immortality obsession and his Mormon background who have this whole thing about genealogy and eternal families etc.
Or maybe the people who do so aren't actually atheists. There are plenty of atheists out there who don't do that.
That's not accurate at all. AI research has been around since the 1950s and pioneers of the field identified risks early on, including Alan Turing, Norbert Wiener, and I. J. Good.
The problem with Yudkowsky is that he lays out elaborate doomsday scenarios with extreme confidence, except none of it is grounded in realistic physical constraints, timelines, or empirical data. It's all divined a priori from Yudkowsky's ad hoc "rationalist" principles.
>The guy was saying that AI was going to be a serious problem decades before anyone else was even considering it as a remote possibility. How is that in any way "consistently wrong with all their predictions"?
I was mainly responding to the first part of that. If we're talking about his track record, we should look at the specific predictions he's made.
https://www.longtermwiki.com/wiki/E643
Some that turned out to be wrong include: singularity by 2021, singularity by 2025, 70% chance of human extinction between 2003-2015, his team would build "final stage AI" reaching transhumanity between 2005-2020, "probably around 2008 or 2010." His only correct prediction has been that AI would win gold in the IMO by 2025.
If you have examples of other predictions of his that have come true, I'd be interested to hear them.
I don't think there's much to be gained out of hashing out whether every prediction Yudkowsky has said was right or wrong: I likely directionally agree with you there, as I also find some of his more extreme predictions to be inaccurate. I mostly take issue with "consistently wrong". The results in longtermwiki are not "consistently wrong". He's definitely wrong sometimes. But consistently?
People pushed back against his AI doom claims not because the idea that AI could be bad was unthinkable (just look at literally every SF story from the last 50 years featuring AI?), but because he specifically was worried about AI becoming God. That is no closer to happening than it was 30 years ago.
One previous attempt at AI was "expert systems", and while the term fell out of use it's functionally about the same as basically all modern business systems, and doing that wrong led to a lot of people being prosecuted for crimes they didn't commit: https://en.wikipedia.org/wiki/British_Post_Office_scandal
If you insist on learning systems, all AB testing counts, and from that we get meta making their products into hyper-stimuli for vulnerable teens: https://finance.yahoo.com/markets/stocks/articles/meta-meta-...
Even without that lawsuit, there have been various concerns that "the algorithm" (of Twitter/X, FB, Instagram, YouTube, Google, everyone) has propagandising biases that damage society. I don't know how to sift fact from politicking with that.
Sounds like corporations can have bad motivations, and can make bad software - no AI required.
"it's functionally about the same as basically all modern business systems"
All AI *is software*; and by the old definition of AI, almost all modern software *is AI*.
Besides what kind of slippery slope argument is this? All software is AI, so if you do harm with software, AI is a harmful technology? Seems like a not very helpful model of the world.
If I scream that a vicious beast is going to destroy the world, I don't get credit for being 'directionally right' if a squirrel eats a hazelnut. AI being a big deal is a long, long way from the full beliefs Yudkowksy promotes, and there are many people who predicted AI's significance who didn't also believe it would be an extinction event.
It’s more he screamed that bacteria is gonna do evolution faster than any before it and turn into a beast that destroys the world, and now that bacteria has turned into a squirrel so far. He’s been very right so far and if that squirrel keeps sizing up you’d want to give him credit.
Just recently AI became “powerful enough” to warrant export controls from the US and maybe soon China.
Someone makes a made up claim, second party acts on made up claim, third party takes second party’s actions a proof of claim.
Reminds me of the “they’re eating pets” thing a bit.
Like are you saying that AI is still kinda not a powerful tool and unlikely to become geopolitically relevant etc? I don’t know how to relate this to culture war media frenzies which is rather stay out of.
It was a topic in less-mainstream sci-fi well before that. And some more mainstream stuff like Star Trek TOS.
Frankly it seems more common than not in the last 40 years. I don't really remember a big wave of claims "Terminator is silly, no sort of AI could ever be malevolent!"
We've even had the dichotomy of some Doctor Who episodes where the self-driving car crashes itself to kill the occupant to silence them, against repeated real-life news stories about self-driving cars killing people, and yet the connection isn't getting made that software controls hardware.
Only if it was an uncommon prediction; otherwise, it’s just evidence of common sense.
IMO they have been consistently too aggressive on timelines. When GPT-4 came out, Yudkowsky said it might be conscious. I think he has written interesting stuff but let's be real
I don't expect GPT-4 to be conscious, it would be quite remarkable if we've stumbled upon whatever evolution did that made us conscious.
But how would you even tell, given we don't know the mechanism or physical process that un-thinking and un-planning natural selection happened to stumble upon to give it to us?
Stories about golems going rogue and genies requiring careful wishing aren’t modern era inventions.
Doom is not a logical outcome given the current human condition.
Sorry, but most times I've talked to someone who says this (about AI completely replacing humanity), they don't have anything to say about why it's unlikely to happen other than:
- "well, it's just such an extreme outcome, it must be improbable"
- "humanity has survived near scrapes with extinction before"/"all the previous doomsday predictions have been wrong"
There is a cliche that all teenagers deep down believe that they are invincible. It seems to me that humanity is still a teenager in this respect: We don't take seriously the possibility of our own extinction. While one might think that the invention of nuclear weapons would serve as a wake-up call, if anything it has done the opposite.
I'm willing to hear arguments besides the two above, if you have them. (And to be clear, being replaced by AI doesn't necessarily mean being replaced by LLMs in particular. They are a relatively new development.)
I sorta gather (just a hunch) this is the type of device Open AI is working on.
Though if there are no websites anymore or ones not many visit how would AI stay relevant? I wrote some thoughts about this in early June https://ryanspahn.substack.com/p/ai-to-pay-for-all-americans... that AI needs to pay it's fair share for all it has taken and all it will continue to take from us. A symbiotic relationship needs to be established!
Also, in this model Cloudflare basically becomes the centralized gate keeper of you getting paid. And to me, this kind of sucks. It's not a very productive line of thought on my end -- I could see something like this happening as the internet trends toward centralization over time -- but it just feels bad.
Though this human would be fine if the universe just unplugged AI and I could go back to working as a well paid UX Engineer/Designer where jobs were plentiful and it was easy to get a job (not have to go through 15 interviews). Thanks AI, but as a startup person I am deep into vibe coding now all my crazy ideas and without having to lean on others (so far). That's pretty cool!
That's what the robots are for.
I am not expecting the humanoid robots to meet the hype for a long long time*, but even boring industrial robots (CNC machines), even boring commercial robots (vending machines), even boring household robots (lawn roombas) have made incremental changes even though we don't yet have an AI good enough to be general purpose over them all.
* for power-envelope reasons alone there should be a ten year gap between "self driving car of quality X" and "humanoid robot which can get into normal car and drive it at quality X".
In the current media ecosystem, what gets you "social cache" or whatever? Let's call them "cool points." Well, right now, the current metric is basic "likes" or "retweens" or "upvotes" or whatever. And to get those, you have to make a claim and then that claim has to be evaluated by others. But the evaluation by others is not really reality, it's just what others think, and it's based on outcomes.
This gives us 4 possible prediction-outcome scores.
You predict DOOOOOOOM!, and the outcome is DOOOOOOM! - in this case you look like a genius because you predicted doom
You predict DOOOOOOM!, and the outcome is "ok, or even great!" - nobody cares you got it wrong because things are fine
You predict UTOPIA!, and the outcome is "ok, or even great!" - people say, "wow that was cool, you got it right" but its not that great of a reward from your social group (the internet) because things were fine
You predict UTOPIA!, and it turns out DOOOOOM! - you look like a moron.
Now looking like a moron is is wayyyyyy worse than what happens if you get it right. Indeed, unless you are really damn sure of utopia, you have a social incentive to predict DOOOOOM! that's greater than the alternative. You can predict DOOOOOM! 100 times and get it wrong 99 of them and nobody cares because that one time you get it right you look like a genius. It's a huge huge problem in the current media landscape and it's why everything is killer robots right now and not "hey we could build a star trek utopia!" It's effectively a selfish NIMBY-ist view of the world in a way, "why build anything, it's just going to ruin everything!" It's a pessimism that literally desires the end of humanity to "prevent the likely future sufferings." It's a simplistic view of a complicated world, and the rewards that go to "DOOOOM!" don't match much with reality.
That's why the internet is so doom pilled, that's why everyone is a cynic, that's why everyone is kind of an asocial asshole about new ideas on the internet and even in your friend space.
You guys know you can just "do stuff?" I mean, yeah it takes money and time, but you can allocate that - especially now that you don't have to grind out css for hours and hours. But you can just do things - whatever you want. You've always been able to, but it just got a lot easier in many many domains, and it's going to get even easier than that.
People are going to scream slop and "that sucks" until the cows come home, because that's what is rewarded on the internet these days. Meanwhile, as people we MUST believe that we still have some agency to do stuff, then go and actually do stuff while the haters are sitting on their couches shellacking each other with misanthropy.
And don't even get me started on the loudest voices in the ecosystem right now.
Like, Yudkowsky is a nobody. Not to be a dick, but he's a high school dropout forum guy who got lucky. I'm not saying he's always wrong, or even that I oppose autodidact stuff - indeed I'm always self teaching. But this guy never took a ML class in his life, I doubt he can solve an integral un-aided, I imagine some people here have seen this: https://www.reddit.com/r/badmathematics/comments/2bazyc/comm...
And we're all deferring to that guy like he knows wtf he's talking about? It's the AI equivalent of the self-educated anti-vax mom at the height of the pandemic.
And he's not alone either, there are a lot of people who are uninformed, unknowledgeable, and confidently wrong on the internet desperately trying to peel off your attention for an instant so you can look at their site, their ads, and their content.
The same sorts of things can be said for the Zitrons of the world. He's a hell of an entertainer, and really really funny, but he's consistently wrong on AI stuff. He's got a BA (with honours I guess?) in media something or other. And again, he's really funny, but does that make him qualified to pontificate on matters of AI policy or economics or even to make statements about the technology etc?
Regardless, the misanthropy, the cynicism, and the misinformation that's being shouted about is because being a cynic is more "profitable" than being an optimist when you measure your success more by how many people agree with you rather than actually being right and being able to do things.
In general the rewards for public doomerism seem low, except if you think being public about it can shift the outcomes.
People so ascribe “doooooom” to things that actually aren’t doom but instead are “bad” - but everything is hyperbolic now
I really think we need to stop giving credence to people who have
1) Been consistently wrong with all their predictions
2) Demonstrated an endless spiteful cynicism
Some of these people are very talented in their fields, sure. But malevolent and incorrect should be disqualifying when they talk outside them. You don't want the society they want, and they things they believe in are unlikely to happen.
It would be far, far better to listen to the people who never fell down every misanthropic rabbit hole, rather than the ones who have noticed it this time, but want you to still believe them on every other topic.