We need more kludge in our information networks, not less, to stop aggressive AI. Our hydro dams should be air gapped and only semi-automated. Our self driving cars should not be exposed to the public internet. All of our major services, especially those from the government, should be over SSL.
I agree that the Snowden revelations showed a scary avenue for abuse of power, but the solution is to encourage oversight and to keep elected government officials as responsible for breaches of trust. This is why Bill C-51 in Canada was so scary until they updated it to include more oversight.
We will once the maternity machines start killing our children.
("Humans extinct due to lack of interest. 3D-Go tournament and frisbee toss news at 11h.")
Considering that the "scary sounding" cost is a risk of extinction, the benefits of any technology must be infinite, to outweigh the risk; because the cost of extinction itself is infinite (if we go extinct we lose everything).
In other words, this is not a risk anyone can afford. You don't gamble with extinction, in the same way that you don't gamble with jumping from a very high place: either you know that you won't fall all the way (you have a parachute, etc), or you don't jump.
Note that I believe the whole discussion is pointless anyway because machine learning on its own cannot lead to strong AI and neither can any other technology we currently know of, but the above is just for the sake of putting the right chips on the table.
Example: Considering that there is no pricer higher than dying, the benefit of <leaving the house/taking a car instead of walking/...> must be infinite to outweigh the risk.
But people leave houses all the time, and take cars and planes etc.
That just says something about the ability to calculate risk and suppress the fear of dying, not about the cost of dying.
Stagnation is simply a gamble of a different sort. What if strong AI is what's required to survive our next possible extinction event? I could easily see it - something as simple as spotting incoming asteroids before our current techniques and manpower can, for example. If we risk ending civilization a few years early for the chance at extending it a few more millennia, is that worth it? Depends on the chances and your priorities.
> either you know that you won't fall all the way (you have a parachute, etc)
Parachutes fail - this is a gamble. Some people think parachuting worth doing anyways. And if there's a wildfire right behind you, depending on the height, you might risk a high jump even without a parachute.
Known colloquially as "in the long term we're all dead". However, you don't kill yourself because you'll die eventually, neither do you stop fearing death because it's inevitable.
Which means that although you don't know the point in time when you'll "eventually" die, you do everything you can to avoid dying at any time there is a chance that you might.
You do this because you wish to minimise the chances of dying at all costs, and push the time of death as far away as possible. That you are willing to pay any cost to keep on living, regardless of how much longer you have to go, makes the cost of dying "infinite".
If you want to be very precise about it, the cost of dying is not infinite, literally, but it's the highest cost you'll ever pay and you can never profit from your own death, so it might as well be infinite. There is no risk that surpasses it and there is no profit that justifies it.
That's how this "infinite cost" thing works.
>> Parachutes fail
The point is that a parachute would give you a chance of not dying. Without a means to survive a fall, you are not "risking" anything- you are simply jumping to your death.
Or, in other words, a "gamble" is a gamble only if you have a chance to win. Otherwise, it's just charity.
I'm (poorly) trying to point out this is a false premise. If I have the choice between 10 years of bliss, or 11 years of hell, I'd choose the former.
While things are rarely quite this clear cut in real life, some risky-but-profitable behaviors work out this way - where, statistically speaking, the decrease is pretty much guaranteed. I don't have the best diet - because I want to enjoy my meals. I don't have the best exercise regimen - because I have other things I want to do with my time. I'm assuming these will shorten my lifespan, but choose to continue down this path anyways - as I very much feel I am profiting (in merely the enjoyment of my life!) by accelerating my future death.
The cost is high enough that I do pay some attention to my diet, and try to make sure I'm at least walking to lunch regularly as a means of exercise, but nowhere near so high as to make it my singular focus in life. I'm not doing everything I can - not even close - as I would assume I'd do were the cost "infinite", even if we're not speaking literally.
> If you want to be very precise about it, the cost of dying is not infinite, literally, but it's the highest cost you'll ever pay and you can never profit from your own death, so it might as well be infinite. There is no risk that surpasses it and there is no profit that justifies it.
If this were true, even if only just in the moment before one's death, nobody would ever sacrifice themselves for a cause or a loved one. Depression would claim no lives. Religions would condemn their gods for giving us finite lifespans. Medical research would be a much higher priority.
> That's how this "infinite cost" thing works.
Which is exactly why I'm arguing that the cost is, in fact, not infinite. Perhaps you disagree with how other people choose to value their own lives, but I suspect even then you'd find yourself to have a breaking point - where the costs simply aren't worth extending your life.
Assuming you're not a sociopath, you're probably not willing to, say, murder an entire room full of people just to extend your life another hour - even if you only have minutes left. And I suspect even sociopaths wouldn't be willing to, say, be the victim of significant torture for the entire duration of their extended lifespan, to achieve the same. Because the cost isn't infinite, and in such scenarios, is in fact outweighed by the cost it'd take to stay alive.
I'll leave it to you to decide where the break even point is for yourself. But I suspect you'll agree with some advice: Enjoy life! Don't sacrifice all happiness just to extend things a little - the price isn't worth it.
My argument - same as it has been since the Snowden revelations - is that if we cannot roll back this radical shift in the ways of the world, then the only check on the power of governments, corporations, and other potentially nefarious groups, is to have all of this information available to the public at no additional cost.
Anything less is an open door to abuse, corruption, and tyranny.