And what if the creators' requirements don't align with the requirements of broader society? We already know that some nation states can behave relatively responsibly with nuclear weapons but we maintain nonproliferation policies with nuclear weapons to prevent rogue nation states and terrorists from wreaking havoc on the world.

The destruction that nuclear weapons can cause is nothing compared to the potential destruction from self-replicating machines and it is very unlikely that non-proliferation of self-replicating machines will be impractical if not outright impossible.

I am fascinated by youtube channels like Primative Technologies and I am struck by the fact that once self-replicating nanotechnology is developed it is very likely that a properly motivated and well-read individual could walk into a forest with little more than the clothes on their back and walk out with a self-replicating machine, or rather float out with whatever sort of flying machine that their self-replicating nanobots would enable them to build.

Once this technology has been invented there's no stopping it. I don't know what we do then.

[1] https://www.youtube.com/channel/UCAL3JXZSzSm8AlZyD3nQdBA

So first I made a fire drill and start a fire, then dig out some clay and fire it into bricks... *some time later* ... then once the UV laser lithography unit is finished, it's time to assemble the 5nm chip fabrication module for the processing units and Voila. That's the whole thing finished!

I jest, but the thing is just because a machine is capable of self replication, that doesn't mean it's any good at making anything other than itself. Organisms are extremely good at making copies of themselves, but they're ultra-optimised for just that. Even the most sophisticated organisms are only able to make a small set of very crude intentional modifications to their environment, if any. We're the only exception.

Self replicators and omni-replicators are very different things.

Your perspective on self replicating nano bots as well as some other comments I have read here paint a picture of immense change in a short time once the technology is released into the wild. This view seems to ignore the fact that the capability to reproduce and to adapt are not the only factors. Any green and grey goo has to cope with the limitations of it's surroundings. Here are some examples: * available energy sources (food, sunlight, wind, geothermal engery, ...) * the rate of energy conversion into the energy source consumed (electricity or sugar/fats, etc). * processing power (along with it's energy efficiency) * energy consumption/efficiency when interacting with the environment * speed of reproduction * speed of adaptability (through intelligence or evolutionary techniques).

I don't wish to downplay risks involved but even if someone creates an intelligence that can outsmart them or a replicating system which they fear may over replicate the creator should be in control of enough of the environmental factors to keep it under control long enough to reduce their creations access to energy, resources. The real risk (from a perspective of self preservation) is not limiting their creation enough or overlooking sources of energy, information or whatever needs to be limited for this system.

I don't fear a run-away grey-goo scenario where the self-replicators outsmart their creators, I fear that this kind of technology could be used as an unprecedented force multiplier by nefarious individuals or small groups of people.

Nuclear proliferation is like our training wheels for all the dangerous techs that are to come. If you are worried about it, look into how world governance works, how international norms are agreed on, and take a partisan stance against the parties that block the ability to do such things.

Nuclear proliferation: so far so good.

Ozone layer destruction: disaster adverted

CO2 limitations: we would have solved that by now without GWB suddenly deciding that disbelieving in climate change was now a respectable policy position.

> The destruction that nuclear weapons can cause is nothing compared to the potential destruction from self-replicating machines

I think you overestimate the speed at which this will come. We probably could, today, make a self replicating machine from raw materials but it would be the size of a factory and would have a replication cycle of a few months. Going from self-replication to grey-goo will take some time, hopefully we can prepare to it.

And at every step of the short way, people smugly displaying their cool: 'Oh, primitive luddites and their unfounded fear of the wonders of tech. We've been living in Fukushima for 100 years, just build the damn nuclear plant on the seashore already. What could possibly go wrong? A tsunami? That's medieval superstition.'.

Took all of 25 years for a battery powered tablet to be more powerful than a room-sized supercomputer, MFLOP for MFLOP.

https://www.theregister.com/2012/03/08/supercomputing_vs_hom...

Unless you’re suggesting the shirt is itself made of replicating units, I think you’re underestimating the gap between what Primative Technologies et al can do versus the foundation needed to make even a simple microchip.

And given life is self-replicaing nanotechnology, I also think you’re overestimating its limitations, in particular speed.

Just spotted, too late to edit, it should read “overestimating its abilities, in particular speed”.

Yeah personally I feel that it's humans who are the unstable component, unable to keep to policies for millions of years, not the machines. Our weapons have immensely increased in power in the 20th century, and improvements are being researched constantly. The number of countries with nuclear capabilities is ever increasing. How do you prevent nuclear armageddon in such a world? "Don't nuke all of humanity" is a policy that most humans would agree with, but you only need a few who don't. If we want to be a civilization that survives for millions of years, we have to give up control to machines that can uphold such policies for such time spans.

Which is a known bug to over-idealize the unknown. Machines will uphold there programming, but are inflexible and will react wrong (shooting down rockets starting) in the rare situations (large asteroid heads for earth)..

Wait, do you argue that the later a species procreates, and the longer it takes, the more likely it is for long-levity and cancer suppression to co-evolve upwards?