>not the sort of puerile irrationality that privileges Gene Roddenberry's ideas over those of John von Neumann.

Ah yes, the von Neumann that helped developed both the fission bomb and the hydrogen bomb and aggressively promoted the use of them and tried to start nuclear war. I know von Neumann is held in high regard with respect to his role in computing, but he is hardly a guy worthy of respect for his views on the safety of technology.

Certainly von Neumann is far from infallible, but his views on the safety of technology were based on reasoning and knowledge, not what would make exciting prime-time TV. We're literally contrasting the inventor of game theory and modern meteorology* with a TV show where sounds travel through outer space and all the extraterrestrials speak English.

This is like discussing whether Stephen Jay Gould or Jimmy Swaggart has a more credible opinion about evolution. I mean, punctuated equilibrium might or might not be correct, but you're being ridiculous.

https://whatever.scalzi.com/2009/10/13/teching-the-tech/ https://archive.is/khTlp

> when you admit that Star Trek has as much to do with plausibly extrapolated science as The A-Team has to do with a realistic look at the lives of military veterans, life gets easier. ... Meta to this is the discussion of why we have to accept that film/tv SF is riding the shortbus — there’s no actual reason it has to be that way — but let’s not get into that right at the moment.

______

* As well as, as you point out, a significant contributor to the development of the fission bomb, the hydrogen bomb, and modern computation. He was also the guy who axiomatized quantum mechanics and the foundations of mathematics, discovered continuous geometry and quantum logic and Hilbert spaces, and solved the compact-groups case of Hilbert's fifth problem. I guess you don't know much about mathematics, so that won't mean much to you; suffice it to say that the "von Neumann machine" was among the least of his achievements. Not bad for a chemical engineer.

I hadn't noticed this before, but an actual Star Trek writer posted a comment to Scalzi's post:

> Raging at Trek because its filled with rubber science is a straw man argument. When has any Star Trek show ever pretended to be scientifically accurate? Every Trek show has always been, at the core, an action-adventure drama about contemporary issues refelected off the funhouse mirror of an SF setting. It’s allegory, not extrapolation.

So, don't try to predict the consequences of new technologies by analogies to Star Trek. You'll do as well as predicting the outcome of a war by analogies to Bruce Lee movies. Same goes for Terminator, Dr. Who, and the Jetsons.

And, despite https://tvtropes.org/pmwiki/pmwiki.php/Main/MohsScaleOfScien..., the same really goes for fiction in general: fiction is literally, unashamedly, intentionally, nothing but lies. It doesn't tell you anything about reality, just about its authors' beliefs. And when we're talking about things that haven't happened yet, the authors generally don't know any more than you do; even when they have thought about the subject, as is undoubtedly the case with Star Trek's English-speaking extraterrestrials and audio-transmitting vacuum, they may prefer to portray things they know are impossible because they think they'll be more entertaining or help them achieve some other artistic goal.

Don't reason from fictional evidence. It makes you look like a fool.

I am well aware of von Neumann's works and achievements and mentioned just computing because that is what the HN crowd mostly knows him for. But really, do we want to look for ethical advice from a guy who worked on explosives for the military and then moved on to bring about the most destructive weapons that have ever existed? He then went further and helped make them even more destructive. He imo was one of the people directly responsible for the destruction of the islands in the pacific. His morals to me were clearly twisted.

I can still look up to him as an incredibly brilliant mathematician, computer scientist and engineer without conflating that with him being a good and/or wise human being.

I mostly agree about the point about not basing our views on fiction. I just wanted to point out that of all scientists, of which there are many brilliant ones, there are far better choices for sources of ideas on the ethics of technology. Smarts != Wisdom.

> But really, do we want to look for ethical advice from a guy who worked on explosives for the military and then moved on to bring about the most destructive weapons that have ever existed?

We frequently faced with questions of what we ought to do. Should we take Interstate 80 or Interstate 5? Should we eat a high-carbohydrate diet or a ketogenic diet? Should we pursue dating Barbara or Debora? Should we invest in hydrogen bombs or a larger army?

There are right answers and wrong answers to these questions, as well as answers that are in between. Which answers are right and which answers are wrong depends on two questions: first, it depends on which means will produce which results; and second, it depends on which results would be in accordance with our ends, and which would be destructive to our ends.

So, to decide whether to take Interstate 80 or Interstate 5, we must first decide whether we want to go to Santa Cruz or to Sacramento (assuming we are in San Francisco), and then consult a map. The map will tell us whether taking Interstate 80 will have the result of getting to Santa Cruz or not. Then it is of great importance whether the map is accurate and covers the area in question, and of no importance whatsoever that the mapmaker wanted to go to Sacramento, while we want to go to Santa Cruz.

Von Neumann provided an extremely accurate map of the consequences of the hydrogen bomb to the government of his time, as well as the chances of success of various ways of achieving nuclear fusion. The people that followed his advice were able to achieve historically unprecedented military power, which was their intent. If von Neumann's map had been inaccurate, they would have failed and sunken into irrelevance, like our own Project Huemul here in Argentina, which never succeeded in achieving nuclear fusion. The fact that you yourself do not want to create powerful weapons and achieve military victory is of no relevance whatsoever.

The failure of the peaceful nuclear-energy Project Huemul, particularly in the context of the rather heavy bets we had placed on it, was a significant step in Argentina's decline. It was precipitated in large part by the president emptying the academies of his political opponents, depriving his projects of the accurate and brilliant guidance he needed to make good decisions about questions of science. Perhaps, for all their smarts, they were unwise to oppose the president. (This error was repeated by later military dictatorships, further into Argentina's decline, who expelled and sometimes killed the partisans of the banished former president; the University of Buenos Aires, among others, still keenly feels the loss.)

Of course it is possible for a map to be correct about one such factual question and erroneous or silent about another; no map of Argentina will tell you how to get to Bakersfield, and an unreliable map might mislabel Interstate 80 as Interstate 70 despite correctly labeling Interstate 5. But you have given no reason to suspect that von Neumann was inaccurate about the likely results of self-reproducing machines (and, in case you haven't read him, gray goo and Skynet are not similar to what he predicted); you have merely labeled him "unwise" because, at least on questions of nuclear disarmament, he would have been your political opponent.

My understanding is that von Neumann recognized that science would produce these types of weapons eventually, and so pushed hard to reach an equilibrium state of non-use of nuclear weapons under Mutually Assured Destruction (the dude did found modern game theory). So I think it's a bit disingenuous to say that he tried to start nuclear war.

He advocated for a first strike attack (bombing Moscow). Though this was before the Soviets had a strong nuclear arsenal and so likely would not have started a war. So yeah you are right, saying he was pushing for a war is stretching it. He did push for nuking a major city though.

Here’s one from reality, bacteria. If we create self replicating machines, even if we do not intentionally introduce variation, they will vary from generation to generation. They will presumably not be indestructible and so some form of selection will be possible. If their reproduction cycle is fast enough they could potentially deviate from their intended purpose in a very short time.

Remote inspection and checksum of the device firmware can solve this problem. You should worry about intentional hacking of the firmware, i.e. self-replicating botnet.

This assumes that the firmware contains no bugs. As soon as it does, there is always the possibility of unexpected behaviour and an unintended evolutionary trajectory.

I used to be concerned about this issue, due to the obvious analogy with biological evolution, but later I realized that it's an easy risk to guard against.

It's a straightforward engineering problem to reduce the possibility of accidental program mutation to any arbitrarily low level, for example using SHA-2 (with which the chance of an undetected error is 1e-77). Of course, you can have "somatic mutations" where one part of a machine or another malfunctions, for example due to damage or errors during construction; but those don't get propagated to the next generation, so they don't produce the kind of progressive deviation you're describing.

This doesn't happen in nature for a variety of reasons, among which is that as such corrective mechanisms become progressively more perfect, the evolution of the species using them becomes progressively slower, and therefore the perfection of the error-correction mechanisms never quite arrives. Moreover, any species whose evolution becomes very slow is at a major disadvantage when the environment changes sufficiently; it will probably die out and leave no descendants after the next climate change, even something minor like an Ice Age, much less a meteor strike.

Another thing to keep in mind is that the number of generations is quite limited in practice. If, to take an unrealistically risky example, you have an alcohol-dependent nanobot replicator weighing 2 picograms, and you set it to reshaping a tonne of alcohol-soaked soil, it can't make more than 5e17 copies of itself, for which it only needs 59 generations. We aren't talking about thousands or millions of generations: even if time allows for them, space doesn't.

1e-77 (.000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 01) is a number that it can be difficult to get a handle on. If we, to take another unrealistically risky example, converted Ceres into 2-picogram nanobot replicators, there would only be 4.7e35 of them, so the chance that one of them would have a mutation in its program that SHA-2 couldn't detect would be 4e-42, assuming that there were just as many erroneous replications as correct ones. (If only one out of every 1000 nanobot firmware installations had a copying error, the chance that one of them went undetected would instead be 4e-45.) The universe is only 4.3e26 nanoseconds old.

So, human malice or extreme carelessness would be necessary.

In fact, all living things form a category of self replicating machines.

Depending, your [highly optimized] self-replicator might have some stiff competition.

The idea that a designed machine could be competitive with living things at self-replication (the sole target of a multi-millenia optimisation process) seems… pretty unlikely to me.

It doesn’t seem far fetched to me, evolution cannot see nor understand mistakes it makes. There is still plenty of room for optimization.

Grey goo is not so much fiction as it is the logic consequence of the kind of machine its creator "designed" (on the sci-fi definition of designing, but with a very strong focus on science).

If we ever build unsupervised self-replicating machines, we will get grey-goo like problems, that is a completely different threat model from any machine we have ever seen. And if the machines are any fast or any far from us, logic dictates that none of the normal safety measures we apply elsewhere will apply.

(So, I guess the lesson is "don't build unsupervised self-replicating machines". Even our body limits unsupervised self-replication by limiting how many generations a cell can create. But it is surely not "people can misuse any tool", "use it wisely" or "control the consequences".)

> Grey goo is not so much fiction as it is the logic consequence of the kind of machine its creator "designed" (on the sci-fi definition of designing, but with a very strong focus on science).

That's true; I think it's one of the reasons Zyvex's designs didn't include nanobot-sized constructors, instead preferring "convergent assembly" using nanometer-sized parts but not nanometer-sized replicators. Instead, they envisioned that the minimal self-replicating unit would be the size of a laser printer. A person armed with a rock could disable a warehouse full of such devices in a few hours if that became necessary.

So, self-replicating nanotechnology doesn't imply a risk of gray goo.

As a more useful point of reference than space opera and Drexler's thought-provoking speculations, Linux and GCC are capable of replicating themselves, but they don't pose the same risks as the Morris worm.

> that is a completely different threat model from any machine we have ever seen.

Even without the bogus gray-goo threat, that's true of self-replicating machines, but so were my other examples at the time. Well, maybe the PEPCON disaster wasn't so different from the items in https://en.wikipedia.org/wiki/List_of_ammonium_nitrate_disas... and https://en.wikipedia.org/wiki/Largest_artificial_non-nuclear..., in particular https://en.wikipedia.org/wiki/Wanggongchang_Explosion in 01626, which arguably brought about the end of the Ming 18 years later. Nothing like the Wanggongchang explosion had ever happened before, though, and nothing like it would happen again until the 20th century; quite possibly it was the largest human-made explosion before the US started the first nuclear war in 01945, though there are a couple of other candidates.

When facing new technology, fiction is our only guide. We would be fools to disregard the thinking of great minds about the potential, even likely, consequences of new capabilities. Such is the power of modern technology that we must set guidelines on things that do not yet exist, to prevent them from existing. We don't forbid research into genetic weapons because we have experience with them. We ban such things because, in our imagination, we see their potential.

We would be fools to disregard the thinking of great minds like John von Neumann, but even more fools to consider Gene Roddenberry a great mind. For the purposes of prognostication, he wasn't a mind at all; he wasn't even trying to predict what would happen in the future, but to dress up 19th-century Horatio Hornblower naval drama and 20th-century social commentary in whizbang clothing, so people would watch it and he could sell Chevrolets and vitamin pills. See https://news.ycombinator.com/item?id=28519184.