I was interested in Scientology as a microcosm of society around the time alt.religion.scientology appeared in the early 1990s. Scientology documented everything about how it runs, even management is scriptural, and only Hubbard could write scripture so it is like a specimen that's been fixed to a microscope slide.
Since then renegade "Free Zone" Scientologists have put all of his writings online so it is very accessible.
Dianetics is the Sequences of Scientology in that it is a long, windy, and nonsensical book that selects for people who are perseverant readers who are impervious to critical thinking. (Like one of those Nigerian scam emails.)
Hubbard's hypnotic language is demonstrated in his Philadelphia Doctorate Lectures where he not only mentions the hypnotist Milton Erickson but uses Erickson's techniques extensively. (There is no record of Hubbard meeting Erickson but Hubbard and Erickson were both highly active at the same time in Arizona.)
(That is, that weird ramblely kind of writing is by no means innocent but it is misparsed by many people's minds and helps create an altered state of consciousness.)
Hubbard's most satanic (in opposition to conventional Eastern & Western religious ideas) works are Introduction to Scientology Ethics and A New Slant on Life. The first one describes an elaborate system of punishments that are applied to anyone who violates group norms and a system of evaluation that guarantees you will violate those norms by accident. The list of "ethics conditions" is folded so the positive conditions come first and a person who starts reading at the beginning and doesn't go all the way through will miss the batshit crazy stuff. The second describes how a Scientologist is supposed to disconnect themselves from anyone who disagrees with their spiritual journey.
Something Scientology shares with the EY cult (as well as the LaRouche organization and the third international) is the proliferation of front groups. Scientology has Narconon, Criminon, The Way to Happiness Foundation, and many others. The EY cult has "effective altruism", "rationalism", "longtermism", "AI safety" and probably those parties Aella used to run.
It's a typical path that a coercive group makes multiple 360° turns. "Effective Altruism" seems designed to attract morally weak people but seems rational enough at first. I mean, it seemed to make sense when Bill Gates insisted on funding only international NGOs that could prove the value of what they were doing.
Once, however, you are funding "longtermist" organizations that are concerned with imagined problems on the way between here and the glorious future that we've turned all the planetary mass of the Milky Way into Dyson spheres there are no results to measure!
It is like the technique a stage hypnotist uses of putting volunteers through a number of screens that rapidly selects the most suggestible and/or likely to go along subjects. Both in the short term and long term this process expels the trouble makers and leave behind the potential Manchurian Candidates.
It reads like a "Hitler also ate sugar" argument.
It’s true Hubbard divided the world into ‘public’ who would have their bank accounts empty and ‘staff’ who, penniless, would be turned into worker drones.
EY is not building that sort of machine. But Hubbard was operating at peak egalitarianism in the US and EY is operating at a time (and places) that have peak inequality. A Scientology whale gives a few million to the organization, the ideal EY whale is Sam Bankman Fried. Since it is money that is fungible and ‘makes the world go round’ he doesn’t need an army of worked drones when just one rich kid at Stanford or Cambridge can be quietly bled.
Other groups to look at with apocalyptic ideology are the Symbionese Liberation Army (Why make worker drones when you can brainwash Patty Hearst?), People's Temple (...go to San Francisco), Heaven’s Gate, Aum Shinrikyo, etc. It amazes me that one of them hasn’t tried to assassinate an A.I. researcher yet.
>Other groups to look at with apocalyptic ideology are the Symbionese Liberation Army (Why make worker drones when you can brainwash Patty Hearst?), People's Temple (...go to San Francisco), Heaven’s Gate, Aum Shinrikyo, etc. It amazes me that one of them hasn’t tried to assassinate an A.I. researcher yet.
Not obviously bad: have apocalyptic ideology (might actually be a little bit bad, admittedly)
Obviously bad but Eliezer et al. haven't done it: try to assassinate a researcher
It seems EY is now less fashionable, so criticizing him as a crackpot is more acceptable. But I remember a few years ago when he was in full swing, he had a lot of traction even here on HN, and of course in many nerdy circles. Also see that debacle with Roko's Basilisk and all those silly antics. The LessWrong crowd is insufferable.
I can sort of understand why EY had so much pull with internet nerds. He talked the talk, he was confident, and he was a nerd. But I'm also relieved that more people seem to understand now that he is a crackpot, and that claiming one is an expert on AI doesn't make it so, and that writing long essays mixing Bayes, Harry Potter and "rationalism" doesn't really make anyone an expert on anything.
>It's a typical path that a coercive group makes multiple 360° turns.
What is the evidence that Eliezer is running a coercive group? Whom are they coercing into what?
I fully get that LessWrong et al. are obnoxious, use their own terminology, have far-out ideas about the future of AI that are probably wrong, weird moral systems (though any moral system is weird when you look at it hard enough imho)... But you are making the charge that Eliezer is running a cult and doing so on purpose, which is more specific than just running an annoying community around some non-mainstream ideas.
> Some of what appears in it sort of actually makes sense, but when you reproduce it in monosyllables, it turns out to be truisms. It’s perfectly true that when you look at scientists in the West, they’re mostly men, it’s perfectly true that women have had a hard time breaking into the scientific fields, and it’s perfectly true that there are institutional factors determining how science proceeds that reflect power structures. All of this can be described literally in monosyllables, and it turns out to be truisms. On the other hand, you don’t get to be a respected intellectual by presenting truisms in monosyllables.
https://www.openculture.com/2013/07/noam-chomsky-calls-postm...
In contrast, this comment chain was more about EY using large words pretentiously, to suggest expertise or intellectual depth that he doesn't actually have.
And the point is that you don’t need to RLHF as long as you have access to another model that has been trained with RLHF that you can blackbox.
Why does he use 2 different niche analogies to explain the thing that is still quite densely written at the end? Basically I read his point 3 times, and felt it could’ve been distilled into 1 sentence…
“Making profit on AI is basically the same as making profit on other software, except that the sensitive bits include the direct inputs and outputs to the core model, not just the software itself.”
Yes, it could. But then, the attempt to sell the idea this was some kind of radical phase transition that the players in the AI industry (largely, very experienced players in the software industry) simply were fundamentally not prepared for would be a lot harder.
TL;DR: If you allow customers to feed text inputs and obtain the corresponding text outputs from your super-secret-sauce LLM, whether via web interface or API, then your customers can easily and cheaply train their own LLM to learn to act like yours.
For example, spend ~$100 to get input-output samples from OpenAI's text-davinci-003, plus ~$500 to fine-tune LLaMA on them, and you get Alpaca.