News for hackers. Stuff that matters.
Re:Superintelligence: The Idea That Eats Smart People by apsec112 on Thursday December 22, 2016 @05:47PM (#13241088)
This article explicitly endorses argument ad hominem:

"These people are wearing funny robes and beads, they live in a remote compound, and they speak in unison in a really creepy way. Even though their arguments are irrefutable, everything in your experience tells you you're dealing with a cult. Of course, they have a brilliant argument for why you should ignore those instincts, but that's the inside view talking. The outside view doesn't care about content, it sees the form and the context, and it doesn't look good."

The problem with argument ad hominem isn't that it never works. It often leads to the correct conclusion, as in the cult case. But the cases where it doesn't work can be really, really important. 99.9% of 26-year-olds working random jobs inventing theories about time travel are cranks, but if the rule you use is "if they look like a crank, ignore everything they say", then you miss special relativity (and later general relativity).

Re:Superintelligence: The Idea That Eats Smart People by coldtea on Thursday December 22, 2016 @06:35PM (#13241458)
>This article explicitly endorses argument ad hominem

That's because it's very effective in practice.

In the real world (which is not a pure game of logical reasoning only played by equals and fully intelligent beings without hidden agendas), the argument ad hominem can be a very powerful way to cut through BS, even if you can't explain why they are BS by pure reason alone.

E.g. say a person A with IQ 110 talks with a person B of IQ 140. The second person makes a very convincing argument for why the first person should do something for them. Logically it is faultless as far as person A can see. But if the person A knows that person B is shady, has fooled people in the past, has this or that private interest in the thing happening, etc, then he might use an "argument ad hominem" to reject B's proposal. And he would be better of for it.

The "argument ad hominem" is even more useful in another very common scenario: when we don't have time to evaluate every argument we hear, but we know some basic facts about the person making the argument. The "argument ad hominem" helps us short out potentially seedy, exploitative, etc. arguments fast.

Sure, it also gives false negatives, but empirically a lot of people have found that it gives more true negatives/positives (that is, if they want to act on something someone says, without delving into it finely, the fastest effective criterion would be to go with whether they trust the person).

This is not only because we don't have the time to fully analyze all arguments/proposals/etc we hear and need to find some shortcuts (even if they are imperfect), but also because we don't have all the details to make our decisions (even if we have a comprehensive argument from the other person, there can be tons of stuff left out that will also be needed to evaluate it).

Re:Superintelligence: The Idea That Eats Smart People by nohat on Friday December 23, 2016 @12:09AM (#13242984)
It's a reasonable heuristic for when you just don't have the time or energy, but if you are giving a 45min keynote speech on the topic I think you are expected to make the effort to judge an idea on its merits.
Re:Superintelligence: The Idea That Eats Smart People by Nevermark on Sunday December 25, 2016 @01:00AM (#13253159)
Exactly.

The "cultists" he is arguing against are leaders of industry and science. The discourse bar should be extremely high. Way above ad hominem disses.

Re:Superintelligence: The Idea That Eats Smart People by maxerickson on Thursday December 22, 2016 @08:47PM (#13242140)
Einstein didn't look like a crank though. His papers are relatively short and are coherent, he either already had a PHD in physics or was associated with an advisor (I didn't find a good timeline; he was awarded the PHD in the same year he published his 4 big papers).

Cranks lack formal education and spew forth the gobbledygook in reams.

Re:Superintelligence: The Idea That Eats Smart People by rntz on Thursday December 22, 2016 @09:28PM (#13242327)
By this measure, I would say Bostrom is not a crank. Yudkowsky is less clear. I'd say no, but I'd understand if Yudkowsky trips some folks' crank detectors.
Re:Superintelligence: The Idea That Eats Smart People by maxerickson on Thursday December 22, 2016 @09:44PM (#13242407)
Einstein's paper on the photoelectric effect is a bit less than 7000 words.

It is part of the foundation of quantum mechanics.

Superintelligence: Paths, Dangers, Strategies is in the range of 100,000 words (348 pages * roughly 300 words per page).

I'm not familiar with it, but looking around it isn't even clear if it even lays out any sort of concrete theory.

Re:Superintelligence: The Idea That Eats Smart People by philipkglass on Thursday December 22, 2016 @10:14PM (#13242534)
I read Superintelligence and found it "watery" -- weak arguments mixed with sort of interesting ones, plus very wordy.

At the risk of misrepresenting the book, since I don't have it in front of me, here's what bothered me most: stating early that AI is basically an effort to approximate an optimal Bayesian agent, then much later showing that a Bayesian approach permits AI to scope-creep any human request into a mandate to run amok and convert the visible universe into computronium. That doesn't demonstrate that I should be scared of AI running amok. It demonstrates that the first assumption -- we should Bayes all the things! -- is a bad one.

If that's all I was supposed to learn from all the running-amok examples, who's the warning aimed at? AFAICT the leading academic and industry research in AI/ML isn't pursuing the open-ended-Bayesian approach in the first place, largely isn't pursuing "strong" AI at all. Non-experts are, for other reasons, also in no danger of accidentally making AI that takes over the world.

Re:Superintelligence: The Idea That Eats Smart People by rntz on Thursday December 22, 2016 @09:54PM (#13242434)
1. Plenty of academics write books. 2. Comparing a paper and a book for length is obviously unfair. Bostrom has also written papers: https://en.wikipedia.org/wiki/Nick_Bostrom#Journal_articles_... 3. "Concrete theory" is vague. Is it a stand-in for "I won't accept any argument by a philosopher, only physicists need apply"?
Re:Superintelligence: The Idea That Eats Smart People by maxerickson on Thursday December 22, 2016 @10:27PM (#13242592)
I'm not intentionally trying to snub philosophy. With concrete theory, the point I was reaching for is that when you look to measure impact you probably want to point back to a compact articulation of an idea.

The book comparison was probably a cheap shot (on the other side of it, Einstein didn't need popular interest/approval for his ideas to matter; I think that is a positive).

I think as much as anything the comparison is worthless because we can look backwards at Einstein.

Re:Superintelligence: The Idea That Eats Smart People by rntz on Thursday December 22, 2016 @10:36PM (#13242625)
Sure, that's fair. I think Bostrom is no Einstein. But I maintain that he's no crank, either. There's a lot of space in the world for people who are neither.
Re:Superintelligence: The Idea That Eats Smart People by squeaky-clean on Friday December 23, 2016 @11:36AM (#13245600)
Bostrom has also published papers. Comparing a book and a scientific paper isn't very fair.
Re:Superintelligence: The Idea That Eats Smart People by maxerickson on Friday December 23, 2016 @11:39AM (#13245614)
Please see my spirited defense from 13 hours ago:

https://news.ycombinator.com/item?id=13242592

Re:Superintelligence: The Idea That Eats Smart People by squeaky-clean on Friday December 23, 2016 @11:53AM (#13245705)
I saw that, I still disliked the comment enough that I needed to write that. Also to claim that the book has no concrete theory in the same sentence as you stating you're not familiar with the book. Like, c'mon...
Re:Superintelligence: The Idea That Eats Smart People by maxerickson on Friday December 23, 2016 @12:31PM (#13245943)
That isn't what I claimed.
Re:Superintelligence: The Idea That Eats Smart People by erikbye on Friday December 23, 2016 @05:11AM (#13244010)
Bostrom's book is wordy, boring, filled with weak parables, analogies, lacking concretization of a theory.

Like always, alarmist material sells better than realism.

Re:Superintelligence: The Idea That Eats Smart People by grzm on Friday December 23, 2016 @05:23AM (#13244046)
lacking concretization of a theory.

Would you elaborate? It's got some pretty big names recommending it. And Bostrom himself is an Oxford University professor.

Re:Superintelligence: The Idea That Eats Smart People by XFrequentist on Thursday December 22, 2016 @11:10PM (#13242767)
He was awarded the doctorate for one of the papers (the photovoltaic one, if memory serves), after extending it one sentence to meet the minimum length requirement.
Re:Superintelligence: The Idea That Eats Smart People by freshhawk on Friday December 23, 2016 @03:18AM (#13243618)
> but if the rule you use is "if they look like a crank, ignore everything they say", then you miss special relativity (and later general relativity)

No you don't, you just don't catch it right away, relativity actually holds up under scrutiny. Besides, I reject the premise anyway.

Einstein did serious work on the photoelectric effect first and then gradually worked towards relativity. Outside of the pop history he had very little in common with cranks. This is basically the result you end up seeing when you look into any of these examples used to try and argue against the ability to pattern match cults and cranks, the so-called false negatives never (to my knowledge) actually match the profile. Only the fairy tale built around their success matches it.

So it is with cult-ish behaviour as well, these are specific human failures that self-reinforce and while some of their behaviour occurs in successful people (especially successful and mentally ill or far from neuro-typical people) there is a core of destructive and unique behaviour evident in both that you absolutely should recognize and avoid. Not just the statistical argument that you will gain much more than you lose by avoiding it, it's that it is staggeringly improbably that you will lose anything.

Re:Superintelligence: The Idea That Eats Smart People by Sean1708 on Friday December 23, 2016 @09:04AM (#13244729)
Yep, Einstein was an expert in his field who wrote a couple of ground-breaking papers in his field. As far as I can tell no-one who is an expert in AI (or even similar fields) is worried at all about super-intelligence.
Re:Superintelligence: The Idea That Eats Smart People by lern_too_spel on Friday December 23, 2016 @02:52PM (#13246842)
Literally everybody who is an expert in AI is worried about how to manage super-intelligence. The standard introductory text in AI by Russell and Norvig spends almost four pages discussing the existential risk that super-intelligence poses. The risk is so obvious that it was documented by IJ Good at Bletchley Park with Turing, and I wouldn't be surprised if it were identified even before that.
Re:Superintelligence: The Idea That Eats Smart People by linschn on Friday December 23, 2016 @03:15PM (#13246981)
I'm an expert in the field and I'm not worried. It's an industrial risk like any other.
Re:Superintelligence: The Idea That Eats Smart People by Nevermark on Sunday December 25, 2016 @01:19AM (#13253206)
You haven't thought much about the risk of superintelligence if you think it is a typical risk. Is that compared to poorly designed children's toys or nuclear weapons?

I would go as far as to say, "humanity" as it is defined today, is doomed, it is just a matter of time.

The only question is: Will doom play out with a dramatic disaster or as a peaceful handoff/conversion from biologically natural humans to self-designed intelligence.

Either way, natural human beings will not remain dominant in a world where self-interested agents grow smarter in technologic instead of geologic timescales.

Re:Superintelligence: The Idea That Eats Smart People by chc on Thursday December 22, 2016 @06:11PM (#13241261)
That isn't exactly what it's doing. It's proposing that there are two ways we evaluate things — deeply examining and rationally analyzing them in depth to identify specific strengths and weaknesses, and using the very fast pattern-matching "feeling" portions of our brains to identify nonspecific problems. These are cognate to "System 1" and "System 2" of Thinking Fast And Slow.

Having established that people evaluate things these two ways, the author then says, "I will demonstrate to both of these ways of thinking that AI fears are bogus."

Re:Superintelligence: The Idea That Eats Smart People by tikhonj on Thursday December 22, 2016 @08:51PM (#13242159)
It's also a perfectly apt description of, say, certain areas in academia—one that I'm pretty sympathetic too after seeing postmodern research programs in action! Hell, postmodernism is a bigger idea that eats more people than superintelligence could ever hope.

And yet I suspect that many of the people swayed by one application of the argument won't be swayed by the other and vice versa. Interesting, isn't it?

Re:Superintelligence: The Idea That Eats Smart People by to3m on Thursday December 22, 2016 @07:11PM (#13241687)
OK, so I ignore Einstein, and I miss general relativity. And then what? If it's proven true before I die, then I accept it; if it isn't, or if it is and I continue to ignore it, I die anyway. And then it's 2015 and it's being taught to schoolchildren. High-school-educated people who don't really know the first damn thing about physics, like non-hypothetical me, still have a rough idea what relativity is, and the repercussions are.

Meanwhile, rewind ~100 years, and suppose you ignored the luminiferous aether. Or suppose you straight away saw Einstein was a genius? Oh, wait... nobody cares. Because you're dead.

So I'm not sure what the long-term problem is here.

Meanwhile, you, personally, can probably safely ignore people that appear to be cranks.

Re:Superintelligence: The Idea That Eats Smart People by lmm on Friday December 23, 2016 @08:18AM (#13244557)
If it were just a disagreement about physics then it would be safe to ignore.

But in this case, if they're right then we're about to wipe out humanity. That's not safe to ignore.

Re:Superintelligence: The Idea That Eats Smart People by scandox on Friday December 23, 2016 @05:31AM (#13244070)
The argument ad hominem here actually refers to the credibility of the source of an argument. If someone has a clear bias (cults like money and power), then you keep in mind that their arguments are the fruit of a poisoned tree.
Re:Superintelligence: The Idea That Eats Smart People by mcguire on Friday December 23, 2016 @07:22PM (#13247980)
That example is bad, but the arguments aren't quite as objectionable.

"What kind of person does sincerely believing this stuff turn you into? The answer is not pretty.

"I'd like to talk for a while about the outside arguments that should make you leery of becoming an AI weenie. These are the arguments about what effect AI obsession has on our industry and culture:..."

...grandiosity, megalomania, avoidance of actual current problems. Aside from whether the superintelligence problem is real, those believing it is seem less than appealing.

"This business about saving all of future humanity [from AI] is a cop-out. We had the same exact arguments used against us under communism, to explain why everything was always broken and people couldn't have a basic level of material comfort."

Re:Superintelligence: The Idea That Eats Smart People by Nevermark on Sunday December 25, 2016 @01:29AM (#13253229)
>We had the same exact arguments used against us under communism, ...

What nonsense. None of the credible people suggesting that superintelligence has risk are spouting generic arguments that apply to communism or any previous situation.

The question is not IF humanity will be replaced but WHEN and HOW.

Clearly, in a world with superintelligence growing at a technological pace, instead of evolutionary pace, natural humanity will not remain dominant for long.

So it makes enormous sense to worry about:

* Whether that transition will be catastrophic or peaceful.

* Whether it happens suddenly in 50 years or in a managed form over the next century.

* Whether the transition includes everyone, a few people, or none of us.

Re:Superintelligence: The Idea That Eats Smart People by AnimalMuppet on Thursday December 22, 2016 @05:54PM (#13241140)
Er, that wasn't inventing theories about time travel, just about time.
Re:Superintelligence: The Idea That Eats Smart People by apsec112 on Thursday December 22, 2016 @05:57PM (#13241159)
SR and GR explicitly allow time-travel into the future. Which isn't a fully general Time Machine, of course, but is a huge change from 19th-century physics. If SR had just been invented today, and someone who thought it was crazy and didn't know the math was writing a blog post about it, I 100% expect they'd call it "the time travel theory" or some such thing.
Re:Superintelligence: The Idea That Eats Smart People by AnimalMuppet on Thursday December 22, 2016 @06:40PM (#13241505)
> SR and GR explicitly allow time-travel into the future.

I presume you're talking about time dilation here. That's... a little bit true, but not really? At a minimum, it's sure a strange way of looking at it.

Re:Superintelligence: The Idea That Eats Smart People by cloudhead on Thursday December 22, 2016 @07:53PM (#13241897)
It allows wormholes to exist, which can connect arbitrary points in spacetime, hence time travel.
Re:Superintelligence: The Idea That Eats Smart People by state_less on Thursday December 22, 2016 @08:06PM (#13241979)
I time travelled here. What's the big deal? :)

State machines can be checkpoint at any state, load up whatever time you like, is there time per se?

Re:Superintelligence: The Idea That Eats Smart People by TeMPOraL on Thursday December 22, 2016 @09:36PM (#13242371)
Oh guys you talk nonsense. I do dabble in time travel at times, so I know :).

(Why I'm here today? I'm particularly fond of this time period; it's the perfect time when humans had enough computing power to do something interesting, but before the whole industry went to shit. Remind me to tell you about the future some other time.)

Re:Superintelligence: The Idea That Eats Smart People by Confusion on Friday December 23, 2016 @05:25AM (#13244054)
It's not an ad hominem argument if the personal characteristics are relevant to the topic being discussed. The personal characteristics of the people in his example are have empirically been found to be a good indicator of crankhood.
Re:Superintelligence: The Idea That Eats Smart People by Nevermark on Sunday December 25, 2016 @01:41AM (#13253252)
Um, no.

Hawking, Musk, et. al. are highly successful people with objectively valuable contributions who are known to be about to think deeply and solve problems that others have not.

They are as far from cranks as anyone could possibly be.

Anyone can find non-argument related reasons to suggest anyone else is crazy or a cultist, because no human is completely sane.

What someone cannot do (credibly), is claim that real experts are non-expert crazies, over appearances while completely ignoring their earned credentials.

Re:Superintelligence: The Idea That Eats Smart People by Confusion on Tuesday December 27, 2016 @06:36AM (#13262595)
Those were not the people used as an example. The example was real cultists, with robes and rituals that make no sense. And it's not an ad hominem to dismiss them outright based on their appearance. If someone I clearly perceive to be a cultist walks up to me in a mall, I'm not interested in hearing what they have to say, because the odds are empirically large that they will be wasting my time.
Re:Superintelligence: The Idea That Eats Smart People by Analemma_ on Thursday December 22, 2016 @06:14PM (#13241281)
> The problem with argument ad hominem isn't that it never works. It often leads to the correct conclusion ...

You immediately self-contradicted here. If a decision heuristic often leads to the correct conclusion, then it's a good heuristic (if you are under time constraints, which we are).

Of course, given unlimited time to think about it, we would never use ad hominem reasoning and consider each and every argument fully. But there are tens of thousands of cults across the world, each insisting that they possess the Ultimate Truth, and that you can have it if you spend years studying their doctrine. Are you carefully evaluating each and every one to give them a fair shake? Of course not. Even if you wanted to, there is not enough time in a human lifespan. You must apply pattern-matching. The argument being made here isn't really an ad hominem, it's more like "The reason AI risk-ers strongly resemble cults is because they functionally are one, with the same problems, and so your pattern-matching algorithm is correct". Note that the remainder of the talk is spend backing up this assertion.

There's a good discussion of this in the linked article about "learned epistemic helplessness" (and cheers to idlewords for the cheeky rhetorical judo of using Scott Alexander and LW-y phrases like "memetic hazard" in an argument against AI risk), but what it boils down to is that our cognitive shortcuts evolved for a reason. Sometimes the reason is because of our ancestral environment and no longer applies, but that is not always true. When you focus solely on their failure cases, you miss sight of how often they get things right... like protecting you from cults with the funny robes.

> "if they look like a crank, ignore everything they say", then you miss special relativity (and later general relativity).

A lot of people did ignore Einstein until the transit of Mercury provided the first empirical evidence for relativity, and they were arguably right to do so.

Re:Superintelligence: The Idea That Eats Smart People by taneq on Thursday December 22, 2016 @08:55PM (#13242179)
A good heuristic leads to a reduction in the overall cost of a decision (combining the cost of making the decision with the cost of the consequences if you get it wrong).

A heuristic like "it's risky to rent a car to a male under 25" saves a lot of cost in terms of making the decision (background checks, accurately assessing the potential renter's driving skills and attitude towards safety, etc.) and has minimal downside (you only lose a small fraction of potential customers) and so it's a good heuristic.

A heuristic like "a 26-year-old working a clerical job who makes novel statements about the fundamental nature of reality is probably wrong" does reduce the decision cost (you don't have to analyze their statements) but it has a huge downside if you're wrong (you miss out on important insights which allow a wide range of new technologies to be developed). So even though it's a generally accurate heuristic, the cost of false negatives means that it's not a good one.

Re:Superintelligence: The Idea That Eats Smart People by Analemma_ on Thursday December 22, 2016 @09:09PM (#13242235)
I agree with you in principle, but the combination of the base rate for "26-year-olds redefining reality" being so low and the consequences being not nearly as dire as you make out mean I stand by my claim, at least for the case of heuristics on how to identify dangerous cults.

With regards to the Einstein bit, per my above comment I still think that skepticism of GR was perfectly rational right up until it got an empirical demonstration. And it's not like the consequences for disbelieving Einstein prior to 1919 were that dire: the people who embraced relativity before then didn't see any major benefit for doing so, nor did it hurt society all that much (there was no technology between 1915 and 1919 that could've taken advantage of it).

Re:Superintelligence: The Idea That Eats Smart People by Forlien on Friday December 23, 2016 @03:16AM (#13243611)
Pascal's Wager (https://en.wikipedia.org/wiki/Pascal's_Wager) is also about a small but likely downside with a potentially large but unlikely upside. Do you think it's analogous to your 2nd case? If not, how is it different?
Re:Superintelligence: The Idea That Eats Smart People by taneq on Friday December 23, 2016 @07:21AM (#13244394)
Good question, made me stop and think about it!

The difference is that in Pascal's Wager, the proposition is not a priori falsifiable, and so you cannot assign a reasonable expected cost (ie. taking probability into account) to either decision.

In the case of a 26-year-old making testable assertions about the nature of spacetime (right down to the assertion that space and time are interconnected), there's a known (if potentially large) cost to testing the assertions.

Re:Superintelligence: The Idea That Eats Smart People by swolchok on Thursday December 22, 2016 @07:57PM (#13241921)
> If a decision heuristic often leads to the correct conclusion, then it's a good heuristic (if you are under time constraints, which we are).

So, is it a good heuristic to conclude that since crime is related to poverty and minorities tend to be poor, minorities qua minorities ought to be shunned?

Re:Superintelligence: The Idea That Eats Smart People by Analemma_ on Thursday December 22, 2016 @08:10PM (#13241997)
No. The base rate of having a crime committed against you is extremely low, and the posterior probability of having a crime committed against you by a poor minority- even if higher- is still extremely low. My point refers to the absolute value of the posterior probability of one's heuristic being correct, not the probability gain resulting from some piece of evidence (like being a poor minority).
Re:Superintelligence: The Idea That Eats Smart People by taneq on Thursday December 22, 2016 @08:57PM (#13242188)
That actually is a good (in the sense of 'effective') heuristic. It's just not socially palatable in modern, Western civilization.
Re:Superintelligence: The Idea That Eats Smart People by TeMPOraL on Thursday December 22, 2016 @10:09PM (#13242508)
Seconding the point. If you want to accept "ad hominem" or stereotypes as a useful heuristic, you'll quickly hit things that will get you labeled as ${thing}ist. This is an utterly hypocritical part of our culture.
Re:Superintelligence: The Idea That Eats Smart People by curiousGambler on Thursday December 22, 2016 @11:23PM (#13242810)
I've been thinking about this a lot lately, and am coming to the conclusion that it's similar to dead-weight loss in a taxation scenario. As a society we've accepted the "lower efficiency" and deadweight loss of rejecting {thing}ism because we don't want any one {thing} to get wrongly persecuted only on the basis of it being such a {thing}.
Re:Superintelligence: The Idea That Eats Smart People by taneq on Friday December 23, 2016 @01:03AM (#13243170)
If you think of it that way, it's a rephrasing of the old and quite universally accepted "I would rather 100 guilty men go free than one innocent man go to prison."

"I would rather 100 deadbeat {class} get hired than one deserving {class} not be hired due to being a {class}."

Re:Superintelligence: The Idea That Eats Smart People by mcguire on Friday December 23, 2016 @08:05PM (#13248084)
Actually, it isn't.

Let's suppose we want to solutionize our criminal problem. There are 1000 people in the population; 90% white, of which 5% are criminals and 10% black, of which 10% are criminals. (I rather doubt the difference in criminality is 2x, but....)

So, there are 900 white people and 100 black people; if we finalize the black people, we'll have put a big dent in the criminal issue, right?

Well, we reduce our criminals from 55 to 45 while injuring an innocent 9% of the population.

Re:Superintelligence: The Idea That Eats Smart People by GFK_of_xmaspast on Thursday December 22, 2016 @05:59PM (#13241179)
> "if they look like a crank, ignore everything they say", then you miss special relativity (and later general relativity).

Uh ... https://en.wikipedia.org/wiki/History_of_special_relativity#...