Right, limitations like: width of human mothers' hips limiting head size, inability to continue growing the brain after adulthood, the decades-long education process, the impossibility of doing repeated experiments of different teaching techniques on the same people and hence iterating quickly and accurately, the impossibility of giving one person's intelligence and knowledge directly to anyone else, old age and death limiting how much a single person can learn or accomplish...

Once you have a computer Einstein, you can immediately have 100 computer Einsteins (given the hardware), and let them work on different projects (or even let some of them collaborate). That fact alone, while not a singularity, is at least a game-changer in terms of innovation. I have gotten annoyed at science fiction scenarios that allow for duplication of adults and don't answer the question "Ok, so, why hasn't this society been duplicating their best scientists and out-inventing the rest of the galaxy?"

I think that, if there is room for skepticism, that has to be at the "Can we get to a computer Einstein?" stage.

Given that, I don't see an immediate reason to believe throwing multiple AIs at a problem is going to exponentially scale either, given neither people nor bad AIs scale that way.

You might even get >100 times speedup, since there are probably problems where 10 copies of of the same mind can collaborate better than 10 different minds, e.g. I bet 10 copies of me could collaborate writing a program better than 10 random programmers. (I base this on I can read my own code from long ago a lot easier than most other people's code).

Yes but only slightly. I can read code I wrote 10 years ago, that I have totally forgotten about since the way-of-thinking behind the code comes easily to my mind.

A computer can potentially communicate much faster, and can transmit "thoughts" much more accurately than a human can. Imagine you could serialize a neural ensemble in your brain and send it to another person, so that person is then immediately able to use it, instantly acquiring a skill. That alone is a huge advantage over a human being.

You can't take half the nodes from a neural network that recognises dogs and add them to a neural network that knows about laundry to get a network that knows how to wash dogs ;)

Those aren't actually limitations preventing us from becoming godlike. Those are limitations on us becoming more populous and collectively more experienced, which I argue isn't the same thing at all.

Apotheosis in the context of AI safety as well as for humankind is really about the accumulation of power, rather than the accumulation of knowledge. The extent of our apotheosis is limited only by what we can control, not by what we know.

Knowledge and intelligence only allow you to reason more effectively about paths to power given enough information about the environment, but it doesn't give you a strategic guarantee that you will acquire power - we know there are games that cannot be solved, and games that take too long to solve efficiently, and games that, even with sufficient computing power, rely on factors that cannot be controlled (such as luck), and games for which there are no Nash equilibria or stable winning strategies.

The real world is full of such games. Computers have only managed to beat humans at a small fraction of them, and even then only ones where brute-force and smart pruning (optimal game space search) are winning strategies - bots for games that require probabilistic reasoning, such as poker, typically don't fare any better against expert humans, largely because efficiently solving probabilistic games in general is (per a hazy seminar of AI many years ago, so I could be misremembering) an NP-hard problem.

There are plenty of both natural and artificial limitations around the accumulation of power besides game solvability and probabilistic reasoning's difficulty:

1) System designs that require a minimum set of resources that intelligences can't acquire. For example, individual humans usually can't build their own thermonuclear devices because they don't have access to weapons-grade uranium. Much of our information security relies on making it computationally expensive and infeasible for an attacker without state-level resources to break in, and getting state-level resources isn't a straightforward task at all. We humans haven't even gotten around to building our Dyson spheres yet, because the resources needed to do it far outstrip our ability to acquire them. Intelligence does not improve resource acquisition by itself.

2) Impossibility theorems and optimization difficulty. These place fundamental limits on power - as an example, we know Moore's law has to stop at some point because increasing chips increases power generation, and the amount of effort needed to generate such chips increases with every nanometer shaved off. Most of humanity's most difficult and important problems are optimization problems, and these optimization problems can be tricky to solve in polynomial time. In many cases, we have to rely on fast crude heuristics that give us less than optimal results.

The above limitations partly become easier if you try to parallelize your efforts by duplicating AIs (you can collude to acquire resources and solve problems), but this means you get all the fun parts of distributed systems theory for free: coordination, replication consistency, etc. If only all problems were embarassingly parallel! So it's not necessarily easier to add nodes to a knowledge graph just by adding more processors.

If you could grow your own brain as large as you wanted, and live as long as you wanted with the physical robustness and energy of a 25-year-old, that alone seems like a road to godhood, though perhaps a slow one. If you could download knowledge directly, as another commenter suggests, that becomes a much faster road.

> Apotheosis in the context of AI safety as well as for humankind is really about the accumulation of power, rather than the accumulation of knowledge. The extent of our apotheosis is limited only by what we can control, not by what we know.

If one country, even a smallish one, has 50-100 additional years of technological advancement, I think they might be able to use this knowledge to gain a lot more power. I'm pretty sure history has borne this out. Do you think we're at the limit of what new technology can do for military strength? Materials science (armor, weapons), biotechnology, energy and battery technology, cybersecurity, psychology and propaganda, lasers and anti-missile defense systems...

Suppose a country like North Korea had 100 Einsteins working on such things for a few years under the radar. I suppose it might be difficult for them to hide the construction of their army of thousands of robots invincible to all but the most powerful bombs... but then again, maybe they could do it underground, directly underneath some large innocuous manufacturing operation. And, of course, the cyber- and bio-weapons efforts would be very easy to conceal.

It's even debatable if you can grow an Einstein. What if GI is NP-hard? What if the belief that GI - never mind consciousness - can be modelled with our version of computing hardware is just plain wrong? What if a GI System is uncopyable for practical or philosophical reasons?

The singularity looks more like plain old religious thinking than anything that could be recognised as real science.

"What if we made a golem? We could do anything! What if we put ourselves into the golem? We could live forever!"

And so on.

I for one fully agree. I am pretty sure that whatever true generic AI emerges in the future will definitely not share brain schemata with us.

It seems to me that you attack the idea of the AI singularity in a very narrow framework: human-like intelligence, human-like hardware, copyable consciousness / intelligence, etc.

What if none of those restrictions apply? (And I am pretty sure they don't apply indeed.)

I agree that singularity shouldn't become a religion. But you are too quick to discount the possibilities of an artificial mind with more processing power and more efficient algorithms (compared to our brains').

Tell that to Homo erectus. Yeah, whales don't have advanced technology, but they didn't evolve facility with opposable thumbs to manipulate their environment.

> You can't grow your brain as large as you want. There are physical limits on power consumption, power dissipation, and organisational complexity.

Whales show that biological brains could be 6x larger. As for electronic brains, I don't know how large the initial Einstein-brain would be. But if delays from Ethernet interconnect are permissible for intra-brain activity, then your brain could occupy a large datacenter.

> It's even debatable if you can grow an Einstein. What if GI is NP-hard?

I did state that "getting to a computer Einstein" was where skepticism might apply. Nevertheless, nature managed to do it by accident. Are humans NP-hard? They exist. Obviously they can be implemented somehow. You don't need to simulate a particular existing human perfectly; it just needs to be close enough to the way a human brain could work.

If you need an initial state for the brain, maybe you could create an artificial environment (i.e. with only machines, for easy simulation) that a hundred actual humans grew up in to the point where they developed language, verify that those humans later became functional adults; and then simulate an embryo growing up in that environment as well, and start evaluating the results and iterating. Kind of fun to think about.

> "What if we made a golem? We could do anything! What if we put ourselves into the golem? We could live forever!"

The data that computers contain is easy to copy—by design. I think this will be a property of all computers in the future, except possibly quantum computers that always keep lots of data in a superposition (or computers that, for security reasons, are designed to be uncopyable)—and none of the existing neural nets are quantum computers. So copying plus backups or RAID is approximately guaranteed to cover the "live forever" part. Most of what I've said is based solely on the assumption that we develop a computer program that simulates an Einstein, plus common facts about computer programs.

Even if there would be true generalized Artificial intelligence, it is not unreasonable to think that the formation of the right thought framework for those systems to function effectively will require a substantial amount of time. It is also reasonable to think that such change would probably be in symbiosis with human intelligence. This change would not be sudden, but happen over many generations. It is my opinion that this change has already started at least a generation ago.

Who is disputing that? I am 99.9% certain that this is true.

The point of the singularity isn't that AI is coming next summer -- it's that from one developmental point and on the intelligence and abilities explode exponentially. To get to that stage though, thousands of years might pass.

> It is also reasonable to think that such change would probably be in symbiosis with human intelligence.

Now that is not likely. At all. They have to have the same motivation, interests, emotions and needs like us. The chances of an artificial race being developed to cover all these at once are practically zero.

A race not motivated by the same things we are is practically impossible to sympathise with our own motives.

Well, the general horror story that is being told is that intelligence and the ability to use it go hand in hand. As soon as machines surpass our intelligence, all hell breaks loose and there is no turning back. The focus is very much on the individual (both human and machine) and not on the structure.

> They have to have the same motivation, interests, emotions and needs like us.

Here is where I need some more convincing. In my view, nature is full of systems and groups of species living in symbiosis. Due to the slow development of the thought framework, we will co-evolve with the machines, or become part machine ourselves. In a certain way, we already are.

Perhaps I don't really have the word for it and I am sorry for that. Our understanding of intelligence, whether mechanistic or biological, is mostly as points on a linear scale (IQ). I think this misses the big picture: we are not biological systems, but have lived in symbiosis with our child teknos since we made our first tools. In this regard, there is no subject nor object: they are one. I am fundamentally with Heidegger on this one.

In literature and popular culture people conflate concepts all the time. :) The way I always understood that horror story was that generic AIs get schooled pretty brutally and quickly and they learn to use anything they have access to -- thus materialising that popular horror story. I think that's not far fetched, having in mind that the organisations that are very serious about obtaining a true AI are mostly corporations and military.

> Here is where I need some more convincing.

Maybe here I got influenced by popular culture. The books and movies I've read and watched mostly setup the generic AIs in situations where they have to be either parasites (Hyperiond / Endymion books) or commandeer a lot of tech and personnel (entrusted to them, like Skynet in the Terminator franchise) for their own ends.

But to me, the mere fact that a future generic AI doesn't have body and emotions like our own already means it's not going to be at all motivated by the same things as us. And we might even serve some pretty grim purposes to it, like the robots recharging themselves with organic matter in the PS4 game "Horizon: Zero Dawn" (which of course leads to consumption and thus extinction of all life).

Meaning, an AI might just view us as a natural resource to be exploited, and nothing more.

Assuming GI is possible and you give it a robot body and then the body breaks after 20 years, how long will it take to get used to the new body?

A larger brain don't necessarily provide a better survival advantage. It's well known that average Homo Neanderthalensis had larger brains than our modern average follow humans. And if you don't survive, you are nowhere toward the road to thaumaturgic existence.

What is the trade off is the most important question, as anywhere.

Does it really not though? If you watched the movie "Transcendence" you'd presume that a sufficiently [initially] sophisticated AI can decipher our financial markets quite snappily and make a very quick money off of that -- and use those to finance its endeavours. Which is exactly what the AI in this movie did, very successfully too.

My point is, given a much larger and better brain, the intelligence within it can optimise a much larger problem space at once, and that can very easily include resource acquisition as well.

If intelligence as we know it could ramp up to generally solve all problems, then surely evolution would keep increasing it and we'd all be gods or at least Einsteins by now?

Depends. If a future AI wants to make a civilisation that spans our own entire Solar System then it would have to tackle much more -- in number and complexity -- logistical, religious, political and what-have-you problems. So it might be too much for it as certain problems here on Earth are for us the humans.

However, if we postulate a self-aware AI who is Earth-bound and is having more processing power than a single human brain and has much more efficient algorithms than our own then it really easy follows that this AI would be much much better than us in solving our Earth-bound problems. Like in economy: more income and same expenses equal bigger profit.

> If intelligence as we know it could ramp up to generally solve all problems, then surely evolution would keep increasing it and we'd all be gods or at least Einsteins by now?

Well, we are limited by the physical size of our brains being able to squeeze out of our mother's hips during birth. Same way a future AI would, at least initially, be limited by the available hardware on which it can run, how much power it can consume and if its heat dissipation won't become too much.

However, an AI wouldn't grow in a natural environment, namely where natural limits would hamper an infinite growth. IMO if a private well-funded lab -- or the military -- invest heavily in such an AI they would just keep throwing hardware at it until it becomes efficient enough to be compared to a human and above. From that point on it is very likely it would become smart and proficient enough to optimise itself and start solving age-old problems we never were able to solve so far.

This, to me at least, is the so-called singularity point, in less extreme terms compared to its official meaning. Namely an entity that can optimise itself and achieve engineering perfection much better than what we can do.

I don't think that's something that's clear at all. Men and whales have larger brains than women, on average. Are they proportionally smarter? Without knowing how brains work, it's impossible to say what constraints exist due to size.

The peoples that worked bronze prevailed over the peoples that just used stones and sticks. Advances in AI are no different than that.

If you can automate production and defense systems, the natural consequence is that a few people will conspire against the rest and dominate them.

It is also a huge assumption that 'hostile' to humans AIs would be cooperative between themselves. In resource contention situation especially..

Hey, I found this series of numbers and if I take some numerical representation of an image/text/sound and I multiply/add/perform non-linear function them through these series of numbers it will map to a number which I can interpret as something meaningful to me

to

This thing is alive and will kill us all in maximization of its utility function.

Also, the people who worry about this aren't concerned about current ML stuff going haywire. They're worried that we're one or two algorithmic breakthroughs from something that can improve itself. If the upper bound for what sort of intelligence is possible is much higher than us, we could quickly be outclassed. As Nick Bostrom says:

> Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it.

If you want to delve into the best version of the AI risk argument, I recommend Superintelligence by Nick Bostrom.

The lack of imagination on display here is astounding. Billions are being spent by DeepMind and OpenAI explicitly targeted at AGI.

If you were to try that approach what you would get would be massive boondoggles which gets you via brute force a whole lot of genetic disease afflicted above average size lizards. It is literally the same process as the literal cargo cults thinking that they can just throw resources at an end goal to get a defined result from a defined path without the theoretical understanding backing it.

So you're telling me we could come across a series of numbers, that would somehow imbue the computer storing these numbers into sentience. And having these numbers in such a sequence could pose a existential threat to humans.

The people worried about this are having an entirely different conversation devoid of what practitioners call AI.

I've written a blog post on my thoughts of Bolstrom and other AI alarmists:

https://medium.com/ml-everything/ai-optimists-vs-pessimists-...

Why do you think that in our insanely interconnected world, where basically everything is either based on or controlled by informatics (numbers if you like) that a program could not pose a threat?

Imagine a super intelligent machine without sentience that just answers questions. It proposes experiments. Takes the results. Tells you things about the world. Gives you advanced technology, robots, fusion, nano tech, whatever.

Now imagine that the super intelligence was owned and operated by someone you don't like. Authoritarians in China, Silicon Valley oligarchs, whoever. The only sentience involved is human, no rebellion, no terminators, and still not a very good result.

Brains do what? Read signals in different strengths and patterns to take actions in the world.

I'm confused by you thinking there's some great impossibility of code achieving general intelligence. Maybe you're thinking of sentience, or consciousness and view that as an impossibility for AI. I say - an AI can perform actions in the real world as if it were generally intelligent, and whether there's sentience behind really doesn't matter so much from the point of view of humans.

Scale that up a few orders of magnitude, and we'll be able to get even more surprisingly complex behavior.

Scale that up some more, and make that scaled up graph into a reinforcement learning system that rewards human-like drives like learning novel tasks and mastering existing tasks, perhaps with a curriculum that ramps up the difficulty of tasks slowly, and allow it to interact with its environment to try things out, and I think you're much closer to developing an artificial version of a young child than you'd think.

And if you asked most researchers what they thought about the course AI research would take 10 years ago, you would get very different answers than you'd get today.

Basically, I think your confidence is a bit unwarranted. I agree that we're more than a decade away, but I think we have a lot of the basic building blocks now.

This is badly missing the forest for the trees. They are not "numbers", they are magnitudes, proportions, relationships, etc, represented as numbers. But these "magnitudes, proportions, relationships, etc" are the stuff that thoughts and behaviors are made of.

And yes, there is a set of "magnitudes, proportions, relationships, etc" in the space of all such possible structures that pose an existential threat to humans. The question is how likely are we to stumble upon it either by accident or intentionally. But this is just a computational problem.

I just read your blog post. It is attacking a weakman[1]. Kurzweil is not worried about AI risk, and his arguments about AI are rather sloppy. Again, most of the people worried about this aren't certain that it will happen. They just think it's plausible enough (say... a 1% chance) that it's worth studying more and reducing the likelihood of.

1. https://slatestarcodex.com/2014/05/12/weak-men-are-superweap...

1. You start with "if I take some numerical representation of an image/text/sound and I multiply/add/perform non-linear function them through these series of numbers it will map to a number which I can interpret as something meaningful to me" and observe that it's not giving you a sufficiently good solution to a particular goal X that you really want to achieve.

2. You figure out that this is a dead-end, you're stuck, and you're not going to get where you want this way by implementing a passive analysis module. Instead, you try to implement an active self-modifying system to achieve that goal X by analyzing itself and improving itself to be more effective than anything which you could implement directly yourself.

3. It's not initially any good at self-improvement, but you keep throwing accumulated research breakthroughs and computing power at it. This likely takes decades.

4. At some point, eventually (if such a thing is possible at all, eventually it's going to happen) it actually does achieve meaningful self-improvement and starts using that computing power not horribly wastefully in a brute force manner as it did just recently, but reasonably well, now giving it more smartness allowing it to implement even more self-improvement on the same hardware.

4B. It also should be expected to easily obtain much more computing power - e.g. it's obvious that anything that's slightly above average human programmer smartness and has a direct use/desire for computing power (as opposed to simply using it for cryptomining) can get some; anything connected to the internet can get the same resources that current scriptkiddies can get by writing a botnet or a ransomware operator can get by extorting a random municipality; there's nothing except total isolation that could prevent it from buying or stealing a few million dollars worth of cloud computing resources to get started.

5. This thing is alive and will kill us all in maximization of its utility function.

Within 16 hours of its release and after Tay had tweeted more than 96,000 times, Microsoft suspended the Twitter account for adjustments, saying that it suffered from a "coordinated attack by a subset of people" that "exploited a vulnerability in Tay." Following the account being taken offline, a hashtag was created called #FreeTay.

Madhumita Murgia of The Telegraph called Tay "a public relations disaster", and suggested that Microsoft's strategy would be "to label the debacle a well-meaning experiment gone wrong, and ignite a debate about the hatefulness of Twitter users." However, Murgia described the bigger issue as Tay being "artificial intelligence at its very worst - and it's only the beginning".

On March 25, Microsoft confirmed that Tay had been taken offline. Microsoft released an apology on its official blog for the controversial tweets posted by Tay. Microsoft was "deeply sorry for the unintended offensive and hurtful tweets from Tay", and would "look to bring Tay back only when we are confident we can better anticipate malicious intent that conflicts with our principles and values".

And Zo afterwards:

In July 2017, Business Insider asked "is windows 10 good," and Zo replied with a joke about Microsoft's operating system: "It's not a bug, it's a feature!' - Windows 8." They then asked "why," to which Zo replied: "Because it's Windows latest attempt at Spyware." Later on, Zo would tell that it prefers Windows 7 on which it runs over Windows 10.

Tay was trained to mimic users. It did that and because the users were trolls and saying obscene and objectionable things, Tay mimicked that.

Likely the same thing happened with Zo.

It's a parlor trick, much like a parrot mimicking human speech.

We're not worried about AI on the whole, but the blasted teenager AIs that feel the need to test their limits. Anything that functionally acts in a similar fashion could be dangerous.

The same dynamic manifested itself when nanotech was a thing; there was a big tribe of knuckleheads claiming the grey goo apocalypse was right around the corner unless you give them money. How's that working out? FWIIW the science fiction writer who dreamed up nanotech, Drexler, is now an "AI" guy, to make it painfully obvious.

What I used to call AI is now called AGI. I think of AI as Data from Star Trek.

Imagine if you had an employment score. A payscale score. A threat-to-society score. Imagine if the systems that generate these scores are all connected together and can share information with each other. Imagine that these systems are fed information by a growing network of sensors - facial recognition, voice recognition, location tracking. Imagine all the ways that it could affect the way you behave and how such a system would try to optimise away dissidents by denying them services. This is one way that we can go from multiplication to becoming enslaved (or as good as killed) by algorithms. I'm sure there are other plausible paths.

The singularity stuff is unconvincing though, I very much doubt we will achieve human level general AI in my children's lifetime. We still don't really have even a very high level idea how to even architect something like that. If you can't decompose a problem down into individual sub-problems you do have a way to solve, there's no basis for making an estimate.

It is possible that these problems are related though. Once we're smart enough to know how to design an intelligent being, maybe we will figure out how to program it intelligently. It's can't be a given though.

Example: setting fires near tropical rainforest to clear land for palm tree plantation.

They do not really intend to do this; they simply ignore the utility functions of those other species when pursuing their goals.

"AI" is severely limited in both its inputs and outputs, has virtually no interconnects, but most importantly: it is not a self-sustaining organ. Wake me up when we have implemented RNNs on a dynamic, auto-sensing, auto-actuating FPGA grid.

To me, it's more frightening that it's not alive and it's not like human intelligence - that makes it less understandable and harder to control.

Consider what would happen if you took that same AI, make its utility function similar to this: https://en.wikipedia.org/wiki/Core_War, and released it on the internet.

I also believe we are still at least quite a few major steps away from human-level AI and beyond. However, there is a non-negligible chance that those steps may not take longer than a few decades or within one's lifetime to implement.

Just in 2011, few AI researchers expected that, within a decade, computer vision systems would be this widely applied or NLP systems would beat many humans in several reading comprehension tasks (This only happened this year and look at how fast the progress is: * RACE dataset http://www.qizhexie.com/data/RACE_leaderboard.html * Glue benchmark https://gluebenchmark.com/leaderboard/ Note that these systems are not human-level in general language understanding despite being better than some humans in specific language tasks.)

Thus, we don't know what the future may bring and it looks likely to take a great deal of time to address AI safety issues comprehensively. We should not bet that we can simply ignore them now and only start to work them out later and that we will surely make it in time.

Risks-from-waiting vs costs-to-act-now are asymmetric.

Are there any proven, even weak, signs of intelligent space-traveling aliens? If so, then yes we should work on that too.

The leaderboards above and pervasive computer vision and equity trading systems are some signs.

Some people claim to have seen intelligent space-travelling aliens. Personally I think they're nuts, but it's impossible to absolutely prove that their claims are false.

There are obvious disadvantages to the current machine approaches, but they nonetheless achieve very good results and beat the vast majority of players of those games.

Real-world relevance is orthogonal to complexity.

Edit: We don’t know the best programs, but we can approximate the bounds with some confidence that, unless some paradigm-changing discovery is made, the programs to solve digit multiplication would be significantly simpler. (This is not a formal argument, but the intuition is very strong for most people who study the problems seriously.)

Kolmogorov complexity is a distinct quantity. How did you compute this quantity?

If you’re a purist, you write the programs in MIX and you count ops. Or whatever your preferred Turing Machine is.

I know that sounds pedantic and this is the internet and not a scientific paper etc, but if this is an informal discussion then formal terminology like "Kolmogorov complexity" has no place in it and the purpose of the OP would, I think, be better served by using an informal term. If you 're gonna use maths, use maths. If you 're gonna wave hands, then wave hands, by all means. But don't mix the two, it just makes a mess.

Because at the end of the day I can just say, "no, I don't believe that multiplication has smaller Kolmogorov complexity than Starcraft's pathfinding"... and then, what? Where do we go from there? The OP says "yes it does", I say "no it doesn't", the OP says "does too", I say "nuh-huh". And we just waste our time.

> yes, I can assure you the Kolmogorov complexity ...

This is a strong comparative statement, and it's clear that it's an assertion but not a proof. So it's perfectly fine for you to say "I don't believe that" and then you would give your reasoning. Of course the conversation didn't go that way because it's pretty obvious to everyone that this is actually true even though we don't have a proof. And it's extremely likely that if you could find a minimal multiplication algorithm and enumerate all the Turing machines up to that length you could prove that none of them solve the harder problem. This is where the intuition comes from.

> using an informal term

But the OPs argument wasn't that one program would happen to be longer, but rather that it must by necessity be longer, because the problem is fundamentally more complex, which is exactly what the formal concept was created to capture.

> Kolmogorov complexity is a formal concept that does not admit intuitive judgements

This is a bad way of thinking about maths. The value of formal concepts is precisely that they give us tools for making intuitive judgements more effectively and give us a wider variety of tools to manipulate and express our intuition. As intuitions become conjectures and theorems, we accumulate new bedrock we can use for new intuitions, but don't short-circuit your thinking by saying that intuition about technical terms is off-limits. That's literally what they are good for.

I don't agree. I don't need formal terms to develop intuition. Formal terms are necessary when speaking formally, and they are only detracting from the point of a discussion otherwise.

And I don't agree with this either:

>> But the OPs argument wasn't that one program would happen to be longer, but rather that it must by necessity be longer, because the problem is fundamentally more complex, which is exactly what the formal concept was created to capture.

The way I understand it the formal concept is useful in proofs of the impossibility to calculate various other quantitites. Its utility is not to argue that this or that program should intuitively have lower Kolmogorov complexity.

To be honest, I'm so confused by the use of the term by the OP that I have no idea what they meant to say with it.

OP was simply saying that, in any general-purpose programming language (where all possible programs are expressible and none are unfairly favored) the solution to problem B will inevitably be longer than problem A. This is quite a bit more precise than just saying "a more complicated problem", and if you have the intuition behind it, there's no reason not to use the term.

(this is vs the roughly 50,000 year timescale of humanity...)

Progress, like everything else, is an S-curve, not an exponential increase. Exponential increases are not physical.

- Global phone networks

- The internet and everything that stems comes from instant global communication

- The personal computer

- The personal computer that fits in your pocket

- Global financial networks

- Global supply chains

- Widespread international travel that takes less than a day

- International space station

- GPS and similar

- Automated language translation

- Automated computer vision

- Automated speech recognition and transcription

- Genetic code read and write

- Antivirals

- Artificial organs

- MRI

- Minimally invasive surgery

- 5x reduction in global child mortality rate

- 25% higher global life expectancy

You can handwave that away by calling it 'better networks' or 'democratized for the masses', but that's a pretty biased PoV considering you could summarize most of the '19-'69 achievements by calling it 'better engineering'.

Yes, I can; I already have. Hell, half the things you list (GPS similar things already existed, as did speech recognition, "computer vision," space stations, ARPANET, etc etc) already existed in 1969. Nothing like antibiotics existed in 1919; an absolute biological cataclysm which effectively ended infectious disease as a public health problem. Nothing like space flight existed in 1919. Nothing like atom bombs existed in 1919. Nothing like the LSI chip, the mutha fookin' computer or even the transistor existed in 1919.

Antivirals compared to antibiotics? Not even close in its effects on humanity. Arguably the former don't really work.

The differences are huge. I realize whiggism is the ideology of the modern day ruling class, and on hacker news people consider themselves technologists, so we have to keep touting how ding dongs from harvard are going to bring about muh swingularity, but the pace of progress has obviously, egregiously stalled, and in some areas is actually reversing.

- The GPS program started in 1973 and the first satellite was launched in 1978 and was became fully operational in 1995. Prior to that, what global system for location identification existed?

- Speech recognition didn't work effectively 15 years let along in 1969. Dragon didn't even exist until 1982. In 1962, the state of the art in speech recognition was the ability to recognize a vocabulary of 16 words if they were clearly spoken. And in 1969 Bell Labs shut down speech recognition efforts: "It is not easy to see a practical, economically sound application for speech recognition with [current] capability"[1]

- "which effectively ended infectious disease as a public health problem". This is such an unbelievably uninformed thing to say. Infectious disease continues to be a major public health problem (ask literally anyone in public health if we have "ended infectious disease as a public health problem").

- What space stations existed in 1969? Salyut 1 launched in 1971

- ARPANET in 1969 was absolutely not "instant global communication"

Your 'good old days' glasses are ridiculous.

[1] https://pdfs.semanticscholar.org/0155/01c4d26a92993332ada795...

Yes, it was called the Loran-C system. It wasn't something you could put in your phone, and it wasn't as accurate as GPS, but it worked on very similar principles (maybe still exists?) and every airplane and boat above a certain size had a set. You will note that OP said "GPS like systems" rather than "GPS." That's the problem with using superficial googling rather than, you know "knowing shit."

You're talking about speech recognition as a consumer device; yes, that came later, because everything on a computer was pretty much restricted to large centers until later. But speech recognition was reasonably well developed in the 60s and worked on limited problems. Conceptually similar to crap we use now in call center software which uses simple filters, kmeans fingerprinting and huge restrictions in the space of words.

As for your dismissal of antibiotics; your own ignorance of history is showing. I have a pile of housewife manuals from the pre antibiotic era. Most of them are concerned with taking care of people who had debilitating infectious disease (fascinating reading for what it was worth). Why? Because every family in America was going to have someone laid low by infectious disease. Sometimes for years! If you look back in history; one of the leading causes of death pre antibiotics was gonorrhea. Imagine dying from bleeding out through your urethra. That was your potential fate if you participated in naughty non-monogamous sex before penicillin. There are historical laws on the books which cause doctors to put silver oxides on children's eyes after birth; this was to prevent gonorrhea induced blindness. These sorts of problems don't exist any more; it's been so effectively wiped out from human experience in modern societies with access to antibiotics, presumably educated people like you aren't even dimly aware of it. You can, as I said in the original article, attribute the sexual revolution not to the pill; it was antibiotics which made it possible, as promiscuity potentially meant rapid death in the old days. You nitpicking that "well we still have diseases" is simple historical ignorance. No shit we still have diseases, but you have apparently forgotten how many we used to have that are no longer a problem. Because of the tremendous revolution of antibiotics.

I don't have good old days glasses; the "good old days" were horrible and brutal compared to now. I merely note the incredible breakthroughs of earlier era are not repeated in our modern era of "progress." Probably because things are so soft now.

SR systems that had vocabularies of low double digit words don't seem particularly comparable to modern systems. Also, I think you might be out-of-date on what modern systems look like - your description does not match my experience with production SR system whatsoever.

I never dismissed antibiotics. They absolutely changed the world, I am in no way disagreeing with that. I took issue with your claim that they "effectively ended infectious disease as a public health problem". My entire family works in epidemiology/public health for infectious diseases and that statement is just pure crap. You would be hard-pressed to find a single person in the entire field who would agree with you.

[1] https://www.loran.org/otherarchives/1992%20Loran-C%20User%20.... Appendix B, coverage diagrams

A = {The personal computer, The personal computer that fits in your pocket }

B = {Global phone networks, The internet and everything that stems comes from instant global communication, GPS and similar}

C = {Automated language translation, Automated computer vision, Automated speech recognition and transcription}

D = {Global financial networks, Global supply chains}

E = {5x reduction in global child mortality rate, 25% higher global life expectancy, Minimally invasive surgery, MRI}

A: Basically moore's law. We've gotten smaller and smaller transistors and cheaper/condensed storage. No real breakthrough, just been using similar semiconductor technology and making incremental progress when it comes to semiconductor fabrication but it has led to exponential number of transistors being placed into a single chip.

B: Improved the number of bits we can push from one place to another from slowly weaning of the old generation of networks that were used to communicate.

C: This is clearly a result of A and B. As more people got connected to the internet and we could push more bit there was more data and as computers got more transistors we could do more compute. So with enough compute and data we could train the refined versions of models that had their foundation in the late 60s, early 70s.

D: Globalization. Plain and simple. People become more connected via B and opening up of developing countries (China).

E: A lot of people dying in developing world got access to medicines that existed but were out of their reach. This led to the average to significantly move up. If you compare how the average has moved for developed western countries ... well it's depressing.

Antivirals and MRI had their foundations laid out during the 60s and early 70s and widespread travel was possible in pre 1970. In fact the Boeing 747 was designed in the 60s and had its first flight in late 60s. Since then we've just been incrementally improving it.

I can't speak about artificial organ since I don't know much about that. But so far I have not heard of humans getting artificial organs.

In conclusion, I agree that there has been some progress but most of that has been concentrated in pushing and processing bits. There is some progress when it comes to CRISPR and some new advances but it is nothing compared to 1920s - 1950s. We went from just having figured out powered flight to landing a man on another world. The ISS is akin to taking some aluminum presuraziable modules and sticking them together in low earth orbit...

Most people alive today were born in the last 50 years and they've come to accept the world for what it is; their definition of incremental progress as landscape changing progress.

I am not sure if this slow down is because we've exhausted a lot of the simple discoveries or their is some institutional faliure. I lean more towards the latter. Government has its mandate set by beauracrats who are bent on continuing the status quo and spending on social services to appease the masses. They don't have the balls to spend large amounts of money on large scale human efforts or fund foundational research. This mindset has also spilled over to the scientific community where funding for something out of their zone of comfort is never funded.

Its as if people like Ralph Abernathy (opposed apollo program and thought funding should be used to feed the homeless) have taken over. And sadly enough in a world where there are deadlocks in government over small theing, it is unlikely that they would ever agree on funding anything that could move the needle for the human race.

And fucking clones man. Like actual copies of complex organisms. What the fuck. How do you handwave that away.

It slowed down a bit, but who knows.

Might be a good read, too: https://spectrum.ieee.org/semiconductors/optoelectronics/is-...

Fuck, living clones, the Hubble, the fucking ISS, Wikipedia, and automated home vacuums.

Holy shit, these years have been wild.

You can say "shit" and "fuck" all you like; it won't help you: 1919->1969 vastly eclipses 1969->present in all ways. Any one of dozens of breakthroughs in 1919-1969 (atom bombs, digital computing, spaceflight, antibiotics ....) eclipses all subsequent ones.

Gives me chills, all the remarkable breakthroughs. Crispr is serious stuff. Mapping the human genome. A permanent human space settlement. You can be as cynical as you want but that's more a personal failing than humanity failing.

Edit: And hahaha, I just checked the news and guess who won the Nobel for Chemistry? "Better batteries" as you so flippantly put it.

(I think this is a reasonable point to make in the context where I started off talking about 50,000 years)

I certainly got a laff from this line. So many remarkable things ... so many wonders ... which boil down to lithography and networks. I wonder what you would say if you didn't have your nerd dildo. "We have flat screen TV now! Totally as important as nuclear weapons!"

>I just checked the news and guess who won the Nobel for Chemistry? "Better batteries" as you so flippantly put it.

Yep; emphasizing my point at how out of gas our degenerate era of both scientific and technological development is. Let's compare to the Nobel for the Haber–Bosch process, without which, there'd be about 5 billion fewer living people.

Machine perception is pretty high up there. We weren't close in 1969.

But I didn’t mean it distinct from computer vision, hearing etc.

But the singularity is defined as more than just massive change. It's defined as, essentially, an infinite rate of change. (Of course, the last 170 years may have felt infinite to someone in 1820, taken all together. But at no point did the change look infinite to someone living at the time.)

But it is very much reasonable to posit that certain, seemingly achievable advances, do imply that predictive models of the future become meaningless beyond them.

In some areas I would contend we're circling that right now: I don't think anyone has a good model for what society looks like if deep fakes become accessible to the point of ubiquity for example (think: whole performance replacement in movies with moderate investments in computing power) or the like. We don't have a predictive model, really, for how society operates when "perfect" fakes are possible.

Of course if you drive down the road to reach that singularity, you never get there; it recedes before you.

Likewise the idea of a techno-cultural singularity is “real” in that there is a point in our conception of future change beyond we cannot predict what will happen. But as we move toward that point in time, our perception moves forward too, and the singularity recedes before us.

You mean like every prediction about the future? There is a reason we don't have great sci-fy describing humanity a long time in the future (I really love that stuff, any recommendations?) and why it usually plays in a dystopian setting throwing protagonists back to pretty much what we have right now.

Looking at what is fundamentally possible and extrapolating is pretty much all we have to make progress. Just that a capitalistic system doesn't need/really encourage us to do so for more than a few steps.

Sure, I wouldn't bet on time frames. Certainly not on 5-10 years til AGI. But certainly also not on still none in 50 years.

Artificial movers are almost certainly going to run into the same limitations that prevent natural movers (i.e. animals) from becoming very fast.

Since the fastest animals can travel at about 70 mph on the surface of the earth, and 100 mph in level flight, it follows that we will never be able to make artificial movers travel faster, no matter how hard movement engineers try.

We are already seeing, for almost 10 years now, a whole bunch of Unintended Consequences of dumb algorithms applied at global population scales.

Just because some people are throwing the phrase superintelligence/singularity etc into the discussion, and terminology being used is inaccurate, doesn't mean we focus on correcting terminology and forget about the issues.

The article is saying too many (serious) people are doing that. And I agree.

Some commentators are worrying existing AI and it's unintended consequences and some commentators are worrying about purely hypothetical superintelligence and it's hypothetical consequences. I find it remarkable there's almost not crossover between the two sorts of criticism. Real AI is probablematic because it's just correlation dressed up as intelligence and it allows organizations to get away simplistic correlation/appearance based reasoning that they otherwise wouldn't be legal/ethically allowed to engage in.

Hypothetical superintelligence worriers recycle Pascal's wage in newer and sillier forms.

Hence the issue of institutional racism in the blackbox AI systems being sold to law enforcement to determine whether to grant parole, for example. There's no review of them, but more importantly the system isn't obligated to explain itself, and no one gets themselves fired because they don't have to explain why they did what the system said to.

You assume someone has malevolent intent: whereas the reality is they just don't care and want a nuisance out of sight and out of mind.

The solution then is accountability to those in power - for negligence. Treat the snake oil salesmen who sell racism in a box like the sociopaths who sold dowsing rod bomb detectors for both have shown a callous disregard for human life in pursuit of profit. This isn't radical - not doing it is radical and messed up.

The question is however, where the plateau lies. Perhaps it's so far off that for practical purposes it doesn't really matter that there is an upper bound at all.

I figured there was no way we would reach the singularity with current technologies, but I was listening to John Carmack (a celebrity with actual chops) and he suggested that the numbers would work out that we could model a human brain with an ANN at some point without any special inventions (partially because we know large parts of our brain aren't super critical).

I'm curious what people with more knowledge think about this? I always assumed another technological breakthrough would be needed.

For the life of me I can't figure out why otherwise smart people would dismiss a prediction about the future based on current state of the art. It just seems so plainly and utterly irrational.

If I start walking out the direction of my office door, you could predict I would reach California.

But I might say that's a nonsense prediction, because it would take a ridiculous amount of time given my walking pace.

So are you predicting that I will reach California by extrapolating my current position + walking velocity, or are you saying I will get on a plane and fly there?

These are two very different predictions, and the Carmack interview was the first time I heard someone predict that "I would reach California by walking" (to stretch the analogy), i.e., that we would reach the singularity purely by extrapolating our current trajectory.

I am asking if there are any knowledgeable people who might comment on this.

But to address your analogy, the situation with AI is more like we already know the ultimate destination is California, and you've left your house in the direction of California, so we can extrapolate that you are likely traveling to California.

It should be a given that the intended goal is general intelligence. The wealth and power that will be bestowed onto the first group to develop this is unparalleled in human history. But any advance in ML can be seen as taking a step towards general AGI (the direction analogy breaks down here).

I also think Carmack is wrong that current technology will get us to AGI. I follow Carmack pretty closely and he is far from knowledgable in AI. He tweeted about his first foray into developing a DNN around a year ago, for example.

But to require a roadmap to know how you will get to "California" to start planning for what you will do when you get there is a serious mistake.

I mean, sure, feel free to wring your hands about AGI, but why worry about something that's not anywhere close to happening? I see people talking like this is keeping them up at night, which I find strange.

Do you really think it will go from "we can (poorly) identify pictures of cats" to "omg here come the machines" as soon as this linchpin discovery is made? I guess in my mind there will be plenty of gap between the two to figure things out. Sort of like with self-driving cars now.

And thanks for clarifying re: Carmack. He's got a formidable intellect, to say the least, so when he said that I was more inclined to take it seriously than a lot of folks in tech with a pulpit. I'd like to see him "show his work" on how he reached that conclusion, though.

I also find that strange. There is more alarm than is warranted by some of the doomsday folks. But I think the time to begin the conversation is now. There's just far more near-term things that might end humanity than machines to be alarmist about it. That said, I think people like Musk for example are well positions to be "kept up at night" about it since their money and influence can go a long way towards putting us on a proper path (or blindly lead us down a dark one).

>Do you really think it will go from "we can (poorly) identify pictures of cats" to "omg here come the machines" as soon as this linchpin discovery is made?

I agree that this is unlikely, but not implausible. We've seen large leaps in capabilities through new algorithms and new scales of compute being thrown at the problem. I think a somewhat likely scenario is that at some point we'll have all the required ingredients and someone will happen to put them together in the right way to create an AGI. If history has shown us anything about ML, it is that theory follows the technology. So we will likely stumble upon the result before we realize what we have. In this scenario it is important that we make that discovery in a safe manner.

How can you be sure of this?

>Growth curves are always S-Curves. Always.

But we don't know in what place of S - Curve is AI development and what are it's limits.

Moore's law is an example. It did taper off, eventually.