News for hackers. Stuff that matters.
Re:The greatest risk of AI is from the people who control it... by chubot on Friday June 02, 2023 @01:00PM (#36167731)
There were 7 parts, and the last one actually addresses exactly the issue we're talking about:

https://aiascendant.substack.com/p/extropias-children-chapte... (and I still recommend all 7 parts)

It's hard for me to respond to the rest your comments, because I simply don't agree with the framing. To me I see a big trap where people read a lot of words, but don't connect those words to reality, which requires action.

If you only manipulate words (e.g. "intelligence" and "AGI" are big ones), then you're prone to believing in and reflecting illusions.

---

You didn't ask for advice, but you say you're not feeling good. I would draw an analogy to a friend who in 2017 was preoccupied with Trump possibly causing a nuclear war with North Korea. There were a lot of memes in the news about this.

Now certainly I couldn't say at the time that it was impossible. But looking back 6 years, I'm glad that I completely ignored it and just lived my life. And that isn't to say it's gone, and not real -- certainly something like that could happen in the future, given the volatile political situation. But it's simply out of my control. (And that's what I told my friend -- are you going to DO anything about it?)

Regardless of what you believe, I think if you just stop reading memes, and do other stuff instead, you won't regret it in 6 years.

I agree with the Substack in that rationalists can be curiously irrational when it comes to examining where they get their own beliefs. (Or maybe examining them in a "logical" way, but completely missing the obvious fact -- like MacAskill writing an entire book about moral actions, while his life's work was funded by a criminal -- a close associate who he vouched for!)

Like EVERYONE ELSE, they are getting beliefs from their reptile brain ("world will definitely end"), and then they are confabulating logic on top to justify what the reptile brain wants.

And you can't really change your reptile brain's mind by "thinking about it" -- it responds more to actions and experiences than the neocortex. It definitely responds to social status and having friends, which can lead to cult-like behavior.

I'd say that humans have a "circuit" where they tend to believe the "average" of what people around them believe. So if you read a lot of "rationalist" stuff, you're going to believe it, even if's not grounded in experience, and there are gaping holes in the logic.

---

The big trick I see a lot of "rationalist" writing is the same one used in the Bostrom book.

1. You state your premises in a page or so. You admit doubt. You come across as reasonable.

2. You spend 300 pages creating entire worlds around those questionable premises. You extrapolate wildly, far into the future. You invent really fun ideas, like talking about a trillion people in the future across galaxies, and what they want.

You overthink it. Other people who like overthinking like what you wrote.

You don't have a sense of proportion that people who have to act must acquire. People with skin the game.

Following this methodology is a good way to lead to self-contained thoughts, divorced from reality. The thoughts just feed on themselves. If you're wrong, and you frequently are, you can just draw attention to some new fun ideas.

Re:The greatest risk of AI is from the people who control it... by chubot on Friday June 02, 2023 @04:01PM (#36170026)
an edit mangled the comment, and it's too late to fix: "he was upset that I wasn't upset" goes one paragraph later