I also consider these posts somewhat alarmist, but just because it is represented as a list of numbers doesn't mean it can't be a serious threat to humanity. Emerging properties is a thing - I mean if you put it that way, insert any tyrant here was just a bunch of atoms, or just a particular organisation of cells, yet they were this and that gruesome.

Why do you think that in our insanely interconnected world, where basically everything is either based on or controlled by informatics (numbers if you like) that a program could not pose a threat?

You don't need sentience to be a threat. Just intelligence.

Imagine a super intelligent machine without sentience that just answers questions. It proposes experiments. Takes the results. Tells you things about the world. Gives you advanced technology, robots, fusion, nano tech, whatever.

Now imagine that the super intelligence was owned and operated by someone you don't like. Authoritarians in China, Silicon Valley oligarchs, whoever. The only sentience involved is human, no rebellion, no terminators, and still not a very good result.

You've described google or probably more askjeeves's eternal promise never realized.

Never realized? You very much underestimate the affect google has had on the world. Imagine how different the world would look if Google and other search engines required a 10k / month subscription to gain access.

Drones are programmed by numbers and take action in the real world. Some ML-based AI analyze sets of data and perform actions in the real world.

Brains do what? Read signals in different strengths and patterns to take actions in the world.

I'm confused by you thinking there's some great impossibility of code achieving general intelligence. Maybe you're thinking of sentience, or consciousness and view that as an impossibility for AI. I say - an AI can perform actions in the real world as if it were generally intelligent, and whether there's sentience behind really doesn't matter so much from the point of view of humans.

Yes, they're numbers/weights. Put a lot of them together in an interconnected graph, and let them adjust themselves to improve their performance on a scoring function, and you can get surprisingly complex behavior. Surprising, at least, if you think of it as "just numbers".

Scale that up a few orders of magnitude, and we'll be able to get even more surprisingly complex behavior.

Scale that up some more, and make that scaled up graph into a reinforcement learning system that rewards human-like drives like learning novel tasks and mastering existing tasks, perhaps with a curriculum that ramps up the difficulty of tasks slowly, and allow it to interact with its environment to try things out, and I think you're much closer to developing an artificial version of a young child than you'd think.

And if you asked most researchers what they thought about the course AI research would take 10 years ago, you would get very different answers than you'd get today.

Basically, I think your confidence is a bit unwarranted. I agree that we're more than a decade away, but I think we have a lot of the basic building blocks now.

>So you're telling me we could come across a series of numbers

This is badly missing the forest for the trees. They are not "numbers", they are magnitudes, proportions, relationships, etc, represented as numbers. But these "magnitudes, proportions, relationships, etc" are the stuff that thoughts and behaviors are made of.

And yes, there is a set of "magnitudes, proportions, relationships, etc" in the space of all such possible structures that pose an existential threat to humans. The question is how likely are we to stumble upon it either by accident or intentionally. But this is just a computational problem.

That's not what I'm saying at all and I've never heard an AI risk person say something like that.

I just read your blog post. It is attacking a weakman[1]. Kurzweil is not worried about AI risk, and his arguments about AI are rather sloppy. Again, most of the people worried about this aren't certain that it will happen. They just think it's plausible enough (say... a 1% chance) that it's worth studying more and reducing the likelihood of.

1. https://slatestarcodex.com/2014/05/12/weak-men-are-superweap...

Studying? From all those years of studying, what has been learnt? I feel like this is deeply stagnant.