That's because it's very effective in practice.
In the real world (which is not a pure game of logical reasoning only played by equals and fully intelligent beings without hidden agendas), the argument ad hominem can be a very powerful way to cut through BS, even if you can't explain why they are BS by pure reason alone.
E.g. say a person A with IQ 110 talks with a person B of IQ 140. The second person makes a very convincing argument for why the first person should do something for them. Logically it is faultless as far as person A can see. But if the person A knows that person B is shady, has fooled people in the past, has this or that private interest in the thing happening, etc, then he might use an "argument ad hominem" to reject B's proposal. And he would be better of for it.
The "argument ad hominem" is even more useful in another very common scenario: when we don't have time to evaluate every argument we hear, but we know some basic facts about the person making the argument. The "argument ad hominem" helps us short out potentially seedy, exploitative, etc. arguments fast.
Sure, it also gives false negatives, but empirically a lot of people have found that it gives more true negatives/positives (that is, if they want to act on something someone says, without delving into it finely, the fastest effective criterion would be to go with whether they trust the person).
This is not only because we don't have the time to fully analyze all arguments/proposals/etc we hear and need to find some shortcuts (even if they are imperfect), but also because we don't have all the details to make our decisions (even if we have a comprehensive argument from the other person, there can be tons of stuff left out that will also be needed to evaluate it).
The "cultists" he is arguing against are leaders of industry and science. The discourse bar should be extremely high. Way above ad hominem disses.
Cranks lack formal education and spew forth the gobbledygook in reams.
It is part of the foundation of quantum mechanics.
Superintelligence: Paths, Dangers, Strategies is in the range of 100,000 words (348 pages * roughly 300 words per page).
I'm not familiar with it, but looking around it isn't even clear if it even lays out any sort of concrete theory.
At the risk of misrepresenting the book, since I don't have it in front of me, here's what bothered me most: stating early that AI is basically an effort to approximate an optimal Bayesian agent, then much later showing that a Bayesian approach permits AI to scope-creep any human request into a mandate to run amok and convert the visible universe into computronium. That doesn't demonstrate that I should be scared of AI running amok. It demonstrates that the first assumption -- we should Bayes all the things! -- is a bad one.
If that's all I was supposed to learn from all the running-amok examples, who's the warning aimed at? AFAICT the leading academic and industry research in AI/ML isn't pursuing the open-ended-Bayesian approach in the first place, largely isn't pursuing "strong" AI at all. Non-experts are, for other reasons, also in no danger of accidentally making AI that takes over the world.
The book comparison was probably a cheap shot (on the other side of it, Einstein didn't need popular interest/approval for his ideas to matter; I think that is a positive).
I think as much as anything the comparison is worthless because we can look backwards at Einstein.
Like always, alarmist material sells better than realism.
Would you elaborate? It's got some pretty big names recommending it. And Bostrom himself is an Oxford University professor.
No you don't, you just don't catch it right away, relativity actually holds up under scrutiny. Besides, I reject the premise anyway.
Einstein did serious work on the photoelectric effect first and then gradually worked towards relativity. Outside of the pop history he had very little in common with cranks. This is basically the result you end up seeing when you look into any of these examples used to try and argue against the ability to pattern match cults and cranks, the so-called false negatives never (to my knowledge) actually match the profile. Only the fairy tale built around their success matches it.
So it is with cult-ish behaviour as well, these are specific human failures that self-reinforce and while some of their behaviour occurs in successful people (especially successful and mentally ill or far from neuro-typical people) there is a core of destructive and unique behaviour evident in both that you absolutely should recognize and avoid. Not just the statistical argument that you will gain much more than you lose by avoiding it, it's that it is staggeringly improbably that you will lose anything.
I would go as far as to say, "humanity" as it is defined today, is doomed, it is just a matter of time.
The only question is: Will doom play out with a dramatic disaster or as a peaceful handoff/conversion from biologically natural humans to self-designed intelligence.
Either way, natural human beings will not remain dominant in a world where self-interested agents grow smarter in technologic instead of geologic timescales.
Having established that people evaluate things these two ways, the author then says, "I will demonstrate to both of these ways of thinking that AI fears are bogus."
And yet I suspect that many of the people swayed by one application of the argument won't be swayed by the other and vice versa. Interesting, isn't it?
Meanwhile, rewind ~100 years, and suppose you ignored the luminiferous aether. Or suppose you straight away saw Einstein was a genius? Oh, wait... nobody cares. Because you're dead.
So I'm not sure what the long-term problem is here.
Meanwhile, you, personally, can probably safely ignore people that appear to be cranks.
But in this case, if they're right then we're about to wipe out humanity. That's not safe to ignore.
"What kind of person does sincerely believing this stuff turn you into? The answer is not pretty.
"I'd like to talk for a while about the outside arguments that should make you leery of becoming an AI weenie. These are the arguments about what effect AI obsession has on our industry and culture:..."
...grandiosity, megalomania, avoidance of actual current problems. Aside from whether the superintelligence problem is real, those believing it is seem less than appealing.
"This business about saving all of future humanity [from AI] is a cop-out. We had the same exact arguments used against us under communism, to explain why everything was always broken and people couldn't have a basic level of material comfort."
What nonsense. None of the credible people suggesting that superintelligence has risk are spouting generic arguments that apply to communism or any previous situation.
The question is not IF humanity will be replaced but WHEN and HOW.
Clearly, in a world with superintelligence growing at a technological pace, instead of evolutionary pace, natural humanity will not remain dominant for long.
So it makes enormous sense to worry about:
* Whether that transition will be catastrophic or peaceful.
* Whether it happens suddenly in 50 years or in a managed form over the next century.
* Whether the transition includes everyone, a few people, or none of us.
I presume you're talking about time dilation here. That's... a little bit true, but not really? At a minimum, it's sure a strange way of looking at it.
State machines can be checkpoint at any state, load up whatever time you like, is there time per se?
(Why I'm here today? I'm particularly fond of this time period; it's the perfect time when humans had enough computing power to do something interesting, but before the whole industry went to shit. Remind me to tell you about the future some other time.)
Hawking, Musk, et. al. are highly successful people with objectively valuable contributions who are known to be about to think deeply and solve problems that others have not.
They are as far from cranks as anyone could possibly be.
Anyone can find non-argument related reasons to suggest anyone else is crazy or a cultist, because no human is completely sane.
What someone cannot do (credibly), is claim that real experts are non-expert crazies, over appearances while completely ignoring their earned credentials.
You immediately self-contradicted here. If a decision heuristic often leads to the correct conclusion, then it's a good heuristic (if you are under time constraints, which we are).
Of course, given unlimited time to think about it, we would never use ad hominem reasoning and consider each and every argument fully. But there are tens of thousands of cults across the world, each insisting that they possess the Ultimate Truth, and that you can have it if you spend years studying their doctrine. Are you carefully evaluating each and every one to give them a fair shake? Of course not. Even if you wanted to, there is not enough time in a human lifespan. You must apply pattern-matching. The argument being made here isn't really an ad hominem, it's more like "The reason AI risk-ers strongly resemble cults is because they functionally are one, with the same problems, and so your pattern-matching algorithm is correct". Note that the remainder of the talk is spend backing up this assertion.
There's a good discussion of this in the linked article about "learned epistemic helplessness" (and cheers to idlewords for the cheeky rhetorical judo of using Scott Alexander and LW-y phrases like "memetic hazard" in an argument against AI risk), but what it boils down to is that our cognitive shortcuts evolved for a reason. Sometimes the reason is because of our ancestral environment and no longer applies, but that is not always true. When you focus solely on their failure cases, you miss sight of how often they get things right... like protecting you from cults with the funny robes.
> "if they look like a crank, ignore everything they say", then you miss special relativity (and later general relativity).
A lot of people did ignore Einstein until the transit of Mercury provided the first empirical evidence for relativity, and they were arguably right to do so.
A heuristic like "it's risky to rent a car to a male under 25" saves a lot of cost in terms of making the decision (background checks, accurately assessing the potential renter's driving skills and attitude towards safety, etc.) and has minimal downside (you only lose a small fraction of potential customers) and so it's a good heuristic.
A heuristic like "a 26-year-old working a clerical job who makes novel statements about the fundamental nature of reality is probably wrong" does reduce the decision cost (you don't have to analyze their statements) but it has a huge downside if you're wrong (you miss out on important insights which allow a wide range of new technologies to be developed). So even though it's a generally accurate heuristic, the cost of false negatives means that it's not a good one.
With regards to the Einstein bit, per my above comment I still think that skepticism of GR was perfectly rational right up until it got an empirical demonstration. And it's not like the consequences for disbelieving Einstein prior to 1919 were that dire: the people who embraced relativity before then didn't see any major benefit for doing so, nor did it hurt society all that much (there was no technology between 1915 and 1919 that could've taken advantage of it).
The difference is that in Pascal's Wager, the proposition is not a priori falsifiable, and so you cannot assign a reasonable expected cost (ie. taking probability into account) to either decision.
In the case of a 26-year-old making testable assertions about the nature of spacetime (right down to the assertion that space and time are interconnected), there's a known (if potentially large) cost to testing the assertions.
So, is it a good heuristic to conclude that since crime is related to poverty and minorities tend to be poor, minorities qua minorities ought to be shunned?
"I would rather 100 deadbeat {class} get hired than one deserving {class} not be hired due to being a {class}."
Let's suppose we want to solutionize our criminal problem. There are 1000 people in the population; 90% white, of which 5% are criminals and 10% black, of which 10% are criminals. (I rather doubt the difference in criminality is 2x, but....)
So, there are 900 white people and 100 black people; if we finalize the black people, we'll have put a big dent in the criminal issue, right?
Well, we reduce our criminals from 55 to 45 while injuring an innocent 9% of the population.
Uh ... https://en.wikipedia.org/wiki/History_of_special_relativity#...
"These people are wearing funny robes and beads, they live in a remote compound, and they speak in unison in a really creepy way. Even though their arguments are irrefutable, everything in your experience tells you you're dealing with a cult. Of course, they have a brilliant argument for why you should ignore those instincts, but that's the inside view talking. The outside view doesn't care about content, it sees the form and the context, and it doesn't look good."
The problem with argument ad hominem isn't that it never works. It often leads to the correct conclusion, as in the cult case. But the cases where it doesn't work can be really, really important. 99.9% of 26-year-olds working random jobs inventing theories about time travel are cranks, but if the rule you use is "if they look like a crank, ignore everything they say", then you miss special relativity (and later general relativity).