> We need to accept that some of these ideas ... are inherently dangerous.

This line of reasoning leads to censorship. It's the same line of reasoning used by every dictator and demagogue as well as China today. Why? The idea of democracy seems dangerous to a king.

Ideas are not dangerous. Ideas are never dangerous. Ideas acted upon might cause danger or harm, but an idea itself never does, never has, and never will cause harm directly.

Open debate of ideas, or the testing of opposing ideas to find the best among them, is necessary for progress... not only to find and establish new ideas, but to question widely held ideas today in the hopes of finding something better. To have open debate--to find the best ideas--you need to listen to ideas with which you might not agree, or which you might find--if acted upon--dangerous.

Yes, some ideas are inherently dangerous. An idea of atoms splitting leads to an atomic bomb, for instance.

Some substances, tools, machines, etc are also inherently dangerous, and can't be made not to be.

Does this mean that we should never use a saw, light a fire, or discuss [insert a controversial topic here]?

Somehow people came up with protocols to work with inherently dangerous material things that lower the risk of a serious injury to acceptable bounds. I don't see why this would not apply to inherently dangerous ideas.

I'd hazard to say that for the last 2500 years quite a few devices were invented in this space, allowing to think much of the previously unthinkable without turning into ravaging monsters or becoming insane. I suspect that we could just continue using these devices (e.g. "freedom of speech", "critical thinking", separation of author's personality form their ideas, etc), instead of running in terror from "dangerous ideas" while swinging a banhammer.

> Yes, some ideas are inherently dangerous. An idea of atoms splitting leads to an atomic bomb, for instance.

You're missing the parent's point. An idea not being acted upon is not dangerous. The idea of splitting atoms is not dangerous and the atomic bomb is not dangerous: An atomic bomb getting built (the idea being acted upon) is dangerous.

Why are full disclosure policies recommended by security researchers? Why is open source healthy for security in general? Why is "security through obscurity" frowned upon? Because it is important to be able to discuss threats in order to better protect yourself from them.

Burying your head in the sand is just a shitty tactic for dealing with danger. Being able to discuss controversial ideas is extremely important, whether it's to better protect yourself from them (eg. extreme-right ideology) or to independently research facts hidden from you by people who don't have your best interests in mind (eg. north korean censorship).

And let's be honest here, from what I've seen of my (left leaning) twitter feed, Sam is being attacked extremely harshly for his "controversial" post. If people here think that the article is wrong, this is a terrible way to prove it so.

Leave the security community out of it. Your rhetorical questions are not at all settled dogma in the community.

> Why are full disclosure policies recommended by security researchers?

This is still a somewhat controversial issue. Coordinated or "responsible" disclosure appears to be advocated by most security researchers. Meanwhile, the full-disclosure mailing list still exists, but the activity on the list is far lower than it once was.

> Why is open source healthy for security in general?

This is also controversial. Open source software has its fair share in-the-wild 0days. The "many eyes" theory only works if there actually are many friendly eyes, and so far, it seems like that is not true. Nobody does this crap for free.

> Why is "security through obscurity" frowned upon? Because it is important to be able to discuss threats in order to better protect yourself from them.

"Security through obscurity" does not come from the "information must be free" community. The phrase comes from the idea that you should assume that your adversary knows everything about the defenses you have set up.

> An idea not being acted upon is not dangerous.

A gun not being fired is not dangerous. A deadly virus that is contained is not dangerous. Ideas are seeds; action is impossible without them. All you're doing is pushing the responsibility one step further; instead of preventing the spread of ideas we have to prevent their effects. Now instead of dealing with the cause, we have to deal with the symptoms.

A meme like "immigrants are the source of all our problems" is a dangerous idea, because of the actions it can directly cause. Preventing its spread is just good mental hygiene.

Cults are another place where ideas are most certainly dangerous; but this time for the individual. So is any religion/ideology people are willing to die for.

Most ideas are not dangerous. Also, I might be completely wrong; if I lived in Galileo's time, I might be claiming heliocentricity is a dangerous idea.

A meme like "immigrants are universally good for our country" is also a dangerous idea.

Beyond what 'scrollaway wrote, I'd add that even granting some degree of danger inherent to an idea, it's only because dangerous is synonymous with powerful, which - in case of science and technology - is usually synonymous with useful.

So the idea of atoms splitting leads to atomic bombs, which - when used on people - are evil, but it also leads to nuclear power plants (and by extension, to fixing current energy problems in an environmentally-friendly way), it leads to new scanning methods in medicine, it leads to improvements in agriculture, etc., and it leads to many similarly powerful spin-off ideas. Hell, it could open up the Solar System for us if people weren't scared shitless of it. Sure we got MAD, and we pointed a lot of nukes at each other, but we survived, and so far we're much better off with the idea of splitting atoms than without it.

And genetic engineering, while opening us up for new kinds of dangers, also has so much more potential than nuclear physics. It has the theoretical capacity of solving most of the problems of humanity, from food supply and environmental damage, to sickness, decay and death. I'd say that the biggest moral hazard here is not pursuing it.

And ultimately, let's not forget that scientific ideas are either good or bad models of the world, and whatever practical danger that comes of it is because of people who go apeshit and exploit achievements of science and technology to further their insanity. So maybe let's focus on restricting the impact of dangerous groups of people. As for the ideas themselves, to quote Eugene T. Gendlin:

  What is true is already so.
  Owning up to it doesn't make it worse.
  Not being open about it doesn't make it go away.
  And because it's true, it is what is there to be interacted with.
  Anything untrue isn't there to be lived.
  People can stand what is true,
  for they are already enduring it.

Ideas being dangerous is one of the reasons that free speech is important. If they weren’t dangerous to someone nobody would bother opposing them.

Having an ethical outlook is a great idea. And it's good to criticise ideas you think are unethical.

The point of the post is that just because a person presents an unethical idea doesn't make that person universally unethical. The point is to separate the ideas from the person presenting them, so that people can feel free to present novel ideas without fearing damage to their reputation if the ideas are unpopular.

> The entire point of the post is to separate the ideas from the person presenting them.

But that wasn't the entire point of the post. The post also didn't like people criticizing life-extending research, when the examples were people criticizing the direction of the research itself.

The post was hazy about what, exactly, it wanted people to stop doing, but it seemed to dislike any criticism.

Whether the "life extension is bad for the environment" argument was a good argument or not, criticism of the ethics of work must be part of free discussion.

The article made the distinction between criticism of an idea and criticism of the person behind the idea. "Of course we can and should say that ideas are mistaken, but we can’t just call the person a heretic. We need to debate the actual idea." It says right there that criticism of an idea is welcome.

Moreover, "heresy" implies that a person becomes heretic - socially despicable - by embracing an idea. This is what the author opposes. And for good reason - embracing the concept of heresy means you're only allowed to think things sanctioned by the current sociopolitical philosophy. And since every society has some amount of batshit insane ideas it believes, allowing for heresy means cutting out the safest / most peaceful way of correcting that insanity.

It's fine to criticise the ethics of the work. Beneficial, in fact. The problem arises when you can't separate the ethics of the work from the ethics of the person doing the work.

When "this is a bad idea, maybe you should stop" becomes "you are a bad person".

It's fine in theory to criticise the ethics of the idea alone, but there can be a limit to that.

If the CEO of a mining company has the "ingenious, un-PC idea" to increase mining profits by lobbying Congress and donating to sympathetic lawmakers to ease EPA regulations so he can dump mercury directly in the river, my ethical argument against the idea is that it damages the environment and the people in the communities, possibly for generations.

If his answer is "I know that, but I value profits more," what is left to discuss about the ethics of the idea? We agree on the facts of the matter.

All that's left is for me to say "I think your moral framework is very different from mine," which is really not that different from my saying "I think you are a bad person."

But can't you see the difference here? There is no argument that dumping mercury in a river is good for society, the environment or anything else. It's only "good" for this companies profits and even the real value there is a question given the impact on most people working there.

But extending human life? Maybe it will be bad. Maybe people will keep having children at the same rate and with no one dying anymore the whole planet is completely full of selfish people who don't care about others of the environment. Maybe only the rich will have access, block everyone else from getting it and control the world by simply out living anyone who is in their way.

But what comparable things do we actually see in history? In every country I'm aware of, once basic survival is taken mostly for granted people stop having kids as quickly and often as they can. In fact these days people often wait so long they depend on science to make birth even possible in their advanced age. And most people have, at most, 2 kids.

Imagine if we lived 10k years. Some people might say "well, lets have our kids right now so we can give up 18 years or so now and then have the rest to ourselves" but I suspect most people would say "what's the rush?". Some would even say "why invest 18+ years to extend my legacy to someone I can't control, can only marginally influence for a decade or two? In 10k years they're bound to solve death all-togher. I'll be my own legacy! And if, after 9,900 years they still haven't done it, I can always reproduce then". Imagine not wondering if there was a Troy, we could just ask the people who lived there. In society, we tend to be safe from some tragedy until most people who lived it are dead then we fall in the same trap again. What if they didn't die?

I say all that to say, it is not remotely obvious that extending human life is in the same category of behavior as dumping mercury in the river to make a few bucks so the consequences should not be the same or even similar.

Sure, life-extension is a nuanced question, but the topic under question is whether we should ever criticize the person presenting an "outside-the-box, maybe un-PC" idea, rather than criticize the idea itself.

My mining company example was much closer to the kinds of ideas that are actually pushed by people in the real world, quite frequently, more so than the article's anecdote about life-extension being criticized.

I'm saying that some people seem to pursue profits in such a way that they clearly have a moral framework that doesn't match mine, and arguing over the ethics of the idea with them is fruitless.

Did whoever it was at Volkswagen who had the bright idea to make cars have a system to detect when they were being tested, and change their emissions accordingly, really need someone to debate with about the ethics of the idea? I don't think that was a case of someone with a simple lack of knowledge of the ethical arguments.

These are the kinds of "innovations" people are coming up with in the real world, more-so than life-extension. People are going to create new apps that allow microloans with Bitcoin, with usurious rates, and market them to poor people. People are going to create dolls that listen to your kids and use the info to market toys to them. People are going to create devices to allow you to monitor your health, and sell the info to drug companies, or even to employers. There are going to be plenty of Bad Ideas where the people peddling them know the ethical arguments against them. At some point, we get to criticize the people coming up with these ideas as well.

On the other hand, people not only drive, but also oppose progress; as a smarter than me person (don't remember who it was) said: Science advances one funeral at a time (it's basically like evolution). If people stop dying (barring accidents), we'll basically stop advancing as well.

Aging eventually impairs the brain. If life extension is possible, it will have to protect pretty much all your organs from aging, and why would the brain be excluded?

Paradigm shifts would become much more rare due to ossification of power structures in intellectual and business arenas.

Again, you're preseenting assumptions. We don't know that things will still work that way if people lived radically longer. You certainly don't know that well enough to shun people who try to find out.

Fearing damage to your reputation if your ideas are unpopular is literally how society works. Unethical ideas aren't just unethical out of the ether, they usually involve harming someone or some group. Putting those ideas forward should come with risk to yourself because it definitely comes with risks to others.

This is a reasonable point in the abstract, but it doesn't account for the current reality in which "Look, I found a heretic" affords such outsized rewards in karma (or social cachet, or the personal satisfaction of having taken down a baddie, or whatever you want to call it).

Fifty years ago, you could say women are too dumb to program computers and still keep your job and be invited to dinner parties, and now you would be fired and shunned. Progress, right? The problem is, that power - the power to destroy careers of people who have done something bad - is not being wielded consistently or sparingly. We're aiming it at misogynists and homophobes, and also their defenders, and also people aren't defending them per se but kinda sound like they are if you only read the headline.

(But the point of the essay is not that Brendan Eich and James Damore shouldn't have been fired; it's that the "Brendan Eich and James Damore should be fired" position has caused a chilling effect on open dialogue.)

Nitpicking here, but your example was somewhat ironic, as 50 years ago, Computer were predominantly programmed by women [1] , so the heretical idea would have been to say that men should be programming computers.

[1] https://www.smithsonianmag.com/smart-news/computer-programmi...

"Programmer" was a data entry job that we automated away. They were basically compiling flowcharts that were written by "systems analysts" doing the work we would call programming.

It's like an "EPROM programmer", which is a tool to make the hardware run the software.

In the old days it was thought that only the implementation of an idea could cause harm. The only evidence I've seen presented that the idea itself is harmful rests upon an assumption it will obviously be implemented or you wouldn't be thinking about it. It's aenthema to a free thinker.

Fair fair but to be clear the one example in this article is calling someone unethical who is working to implement an idea.

"people working on this must be really unethical"

It is not clear or obvious that allowing people to live longer is unethical. Resorting inmediately to an ad hominem is not an appropriate form of debate.

No individual has a moral obligation to debate ideas. However if debate is shut down with fallacious arguments it limits the ability for the idea to progress.

If this happens too often then progress will be slowed. That is the point of this post.

But the argument isn't fallacious and it's not an ad hom...

Extending life expectancy will increase human population and harm our environment and that harm to the environment will harm future generations so individuals working towards extending life expectancy are hurting future generations and that's unethical which makes them unethical. You can disagree with the argument but it's not fallacious.

By that argument all medical care is unethical.

Regardless ideas need to be discussed divorced from the people presenting them. Medical care is no more or less ethical because Hitler is making the argument.

Yes but this isn't about presenting an argument, the one example given in this blog post is about someone working towards a goal. If that goal is unethical and the person is working towards it we should be able to call them unethical. It's not about whether Hitler is saying medical care is good, it's about whether someone working to exterminate the Jews can be called unethical.

Everyone has done something that caused harm at some point. If that is the bar to declare a person unethical then we all are. In this framework you are right but it makes the label useless.

Most people would require action which creates direct (not second order as in this example) harm of a large magnitude before applying the label to a person.

In any case labels mean different things to different people in emotionally charged subjects. Which is why they don't have a place in honest debate.

>Most people would require action which creates direct (not second order as in this example) harm of a large magnitude before applying the label to a person.

That doesn't make any sense. If I directly try to eradicate the Jews I'm immoral, if my actions just have a second or third order effect of eradicating the Jews then I'm not? Most people probably use the direct vs second order distinction when it's themselves who are doing something unethical via second order effects but that's just to save some cognitive dissonance. If I know my actions have second order effects of hurting lots of people and I still do it that's unethical.

>In any case labels mean different things to different people in emotionally charged subjects. Which is why they don't have a place in honest debate.

Reality isn't an honest debate, reality is realpolitik. People are likely to be emotionally charged when they're told that it's okay they're being hurt because it's just a second order effect after all. Learn to deal with that emotion and argue against it, not make posts on the internet opining for something which never existed.

> If I directly try to eradicate the Jews I'm immoral, if my actions just have a second or third order effect of eradicating the Jews then I'm not?

This is a common moral principle; it is found in, for instance, the classical Christian doctrine on homicide, where directly willed killing is (leaving aside war and capital punishment) categorically prohibited, but killing (even when it is a certain result, or as nearly so as practically occurs) incidental to some act with a different end is not categorically prohibited, but judged according to the proportionality of the risked harm of the act with the harm it was avoiding. (Self-defense doctrine in American, and some other, law is ultimately strongly influenced by this principle, though it diverges a bit from it.)

First order effects are easily anticipated such that intent can be assumed. Second order effects are not always obvious even to experts in the field and so require debate and consideration. Intent is unlikely in this case without evidence to the contrary.

Of course If your definition of unethical doesn't require intent to harm then this is a meaningless distinction. Another Reason why labels are unhelpful.

>Extending life expectancy will increase human population

[citation needed]

>harm our environment

[citation needed]

>individuals working towards extending life expectancy are hurting future generations

[citation needed]

We don't know any of those things. You can debate it. Even passionately so, but it's wrong and dangerous to assume your unproven beliefs are facts and label people based on them.

And yet here we're discussing an idea for which you personally believe that someone or some group will be harmed but it is not certain, some of us would say it's not even likely. Some of us would even say it would be a great benefit to society and have the opposite affect you claim.

But because you believe (perhaps even irrationally) that the idea could harm you give yourself, and anyone who thinks as you do, the right to damage people's reputation. History has shown this to be very dangerous behavior.

> just because a person presents an unethical idea doesn't make that person universally unethical.

Reminds me of a Kill the Poor sketch by Mitchell and Webb.

I think the difference is if you consider the /idea/ inherently dangerous, or the potential /outcome/ dangerous.

Yes, it's subtle, but the current shift in the US is towards ideas being dangrous

The criticism quoted wasn't about ethical problems in genetic engineering research. It was about life extension as a goal.

I can imagine a debate about that, but I can't see how it is appropriate to create a toxic environment for life extension advocates.

Apparently these researchers / advocates found the environment so toxic they were motivated to move. Perhaps they were over-reacting but it seems likely something is wrong.

And "move fast / break things" applied to human biology research is entirely an insertion of the commenter, not remotely an implication of Altman's piece.

The idea that "ideas can be dangerous" is a dangerous idea.

> We need to accept that some of these ideas are inherently dangerous.

We created atom bombs. Some say AI can destroy humanity (perhaps Sam himself) but here we are, I don’t think you can stop science.. it’s what makes us human.

You can definitely stop science. There's a reason we don't secretly infect populations with various diseases just to see what happens anymore.

> anymore

and I'd also add "to the best of our armchair knowledge"

And to yours, I'd add "for values of 'anymore' smaller than last 30 years", and kindly remind everyone that US military used to spray pathogens on its own, unsuspecting populace, in order to experimentally explore the areas of bioweapons defense and offense.

yeah honestly puzzled why that's so downed since there are still living person whom did that shit, it is not even one generation past and this place is like "we're the supreme moral force of the world! hurr durr"!

I thought yours a very interesting, worthwhile and thought-provoking comment. I'm puzzled why it's voted down. Maybe too succinct. 3^4 words good, 3^2 words baad?

it's because rule#324 subsection 20 of this site

"usa did nothing wrong"

Certain religions have effectively stopped science at the local/regional scale.

Religion is the anathema to science.

Are you saying ideas you consider inherently dangerous are good candidates for censoring?

> We need to accept that some of these ideas (probably not all - it's always difficult to get on the right side of the line) are inherently dangerous.

ostracizing hypothetical talks is the problem here, not pursuing the unethical goal; just talking about what part of the goal makes it unethical is being shut down (according to the article)

What do you mean by "ethical outlook?" Who doesn't have one? You may not agree with their ethics, but they're there. If you're talking about shared ethics, that's known as morality.

Complex indeed, it's a multidimensional balancing act.

Both technology and ethics can progress and regress across a huge range of concepts. Too much unexamined progress (of either tech or ethics) in one area can lead to terrible outcomes.

Being able to debate and have these conversations is essential to finding the balance.

I'm reading this as complaint about opinions condemning discussion of certain concepts, which is fair. But to ignore that other concepts face the same type of condemnation in China is... amazing.

That's exactly what the blog post wasn't realizing. Throwing out ethics is like throwing out law and order because it inhibits you from doing whatever you'd like to do. Does anyone really think they can stand upright against the winds when the weather turns nasty?

Just like laws, ethics are there for a reason, and the social changes going on are happening for a reason.

It's critical to understand that it's not an intentional simplification. Altman and the rest of the rarified VC-connected people on both coasts are very, very disconnected from what its like to live in the lower or middle classes and especially what it's like to be a part of a systemically marginalized community.