This is the same claim always given by the A.I boosters. Who sees firefighters fighting a fire and decides to just plow through and make matters worse?

Because by "significant minority" what you are saying is that 2 or 3 out of 10 human drivers do this.

Of course, software engineers get treated with kid's gloves, otherwise, everyone just freaks out and cries and cries, because their robotic utopia dreams are meaningless nonsense.

Assess a massive fine. Money is lifeblood of a corpo.

It's just a bit too perfect, no?

One project I worked on was a safety critical system that almost got sunk by an overly-simplistic fault response. The device would reboot if it detected any sensor readings out of range. One day someone put a bag over it while it was delivering concentrated oxygen from the air and it rebooted because the ambient o2 sensor was reading well below what anyone could reasonably expect were the machine not enclosed. The patient had to be bagged while the machine came back on.

You simply can’t predict every emergency situation and you most certainly shouldn’t assume a simplistic single-response model is enough to get the passenger to safety. Humans are really good at inferring correct behavior socially and it would be much better for an AV to watch what other drivers are doing to figure out how to handle unexpected situations. To the degree an AI-driven system is unable to do this it is completely unsuited for use on the road.

And I wouldn’t be surprised if quite a few of those who are have never actually fully understood what that means.

I certainly don’t.

But then, there’s a reason I stay away from life-critical systems.

There is no particular reason why advanced planning, modelling and pattern matching ability should need human-like goals.

Heck, even counting votes would be quite impossible (how much divergence does a copy need before it gets its own vote? What it not all modules/nodes are copied?).

Also, the anti-suffrage women seem to have been erased from history as far as popular discourse goes... guess the Women's National Anti-Suffrage League isn't taught in schools (support amongst women for extending the vote was very lukewarm, becoming a majority only after the law changed).

Because driving requires the ability to understand what other drivers are doing. It requires a theory of mind, which means understanding human goals.

I think instead of flirting with this absurd fantasy of self-driving cars we should be doing more to invest in railroads and other forms of mass transit so that people don't need to drive at all. Europe and Asia are far ahead of North America in this regard.

I'm a transit advocate, but there are some fundamental issues that need to be resolved first, especially in North America.

Of course it is trivially true that we all intend to go from point A to B. But that is almost irrelevant. It is the million micro-decisions on the trip that count.

Without seeing the human, I assess in seconds whether the other driver is unusually high or low-skills, focused or distracted or impaired, polite or rude... and this constant assessment, with human understanding, of (others') human goals and attitudes is a key element of driving skills, and yes this is basically all covered in your exception of needing to account for stupidity of people. That exception is just bigger than you think.

Like chess?

Chess is really just about searching through as much of the space as possible, and you can take a few seconds. And believe me, there's a whole lot more stupid behaviours you can encounter on the road while you have to react in less than a second.

That isn't really an accurate description of how the modern DL bots work. They don't need to reference any database of past games while they play.

As an experiment, try making a chess AI without explicitly giving it the rules of the game and see how it performs.

This reminds me of the kinds of intuitions people had about Chess and Go. In retrospect they seem silly, but it made plenty of sense to them at the time. The fact was that there was a solution that machines could use that humans couldn't use. Naturally, a solution that humans couldn't use was hard for them to anticipate being effective.

Single-track railroad like in airport terminal-to-terminal transport? Fine.

Out on the open rail network? Not even close. And that is a LOT more constrained & predictable than an open road network.

Getting to all cars being software-driven seems as much in the near-mid future as was the idea of having full vehicle-to-vehicle telemetry &communications - i.e., sounded like a great idea until it crashed on the rocks of practicality (and inter-corporate politics?); not likely happening in a relevant time frame.

The optimist believes progress will halt at a convenient moment. The pessimist fears otherwise.

We need true GAI to do this, of course, and the question is: will a "real" GAI be willing to drive a car? How much will we have to pay it?

No we will never have GAI.

---

I know never is a long time, but I can comfortably predict this because there is not only technological problems with GAI, but also philosophical problems. The way we define intelligence is such that if we find or invent something that combats human intelligence we immediately redefine it to exclude that aspect.

All that's needed is someone to achieve GAI at a supercomputer level. From then on, it's just a matter of scaling it down. Your iPhone has more compute power than a supercomputer 30 years ago, after all.

And what's also a way to "cheat" is fusing computers and organic matter. Brain-in-a-vat style, or, for yet another Star Trek reference, Voyager's organic compute network (which IIRC even suffered a pathogen infection at one point). And that is definitely possible in the next 10-15 years. Give Musk's Neuralink enough apes to euthanize and it will get done.

In 30 years we will not have an iPhone-sized device with the power of today's supercomputers. Transistor scaling is in its final stages. We'll be lucky to get 2-3 more process nodes and that'll be it.

Perhaps quantum computing will enable new kinds of inference, then we will need to scale superconductors like we did transistors in the past 75 years, that is possible (not in the next 10-15 years though; 30-40 years is more realistic). However I don’t think this will lead to anything more then amazing statistical inference. People theorized artificial neural networks way back in the 1940s and we already had algorithms for it in the 1970s. I’m not aware of any algorithms, not even theoretical ones that gives us AGI, on any computing device (not even quantum computers).

However this misses my point about AGI. It is not about computing capabilities but theory. AGI will have to evaluate situations that has never arisen before without having data that directly directs it. I don’t see how statistical inference can do that, and as such we will need an entirely new methods of training it. Like I said, we haven’t even conceived how an AGI technology works, there are no algorithms which will work in theory even with unlimited computing power. Contrast this, for example, with quantum computing, which has a bunch of algorithms demonstrating it’s capabilities. With AGI all we have is “perhaps our cars become better at driving them selves”. I’m not buying it.

According to my worldview, humans are just robots built out of DNA, proteins, etc. The idea that machines couldn't do everything a human can do (and more) doesn't make any sense to me.

I actually don’t believe in the superiority of the human mind, I think the word intelligence is actually a scientific distraction. Humans are remarkably—but not uniquely—adapted at communicating with each others and manipulating our environment for our benefit (most of the time; but sometimes to our determent, e.g. by causing a climate disaster). My statement about intelligence was not to share my believe about humanity, but a statement about human believes about our humanity (a theory of mind of sorts). I believe that humans are so human centric that we will always evaluate intelligence in human terms. As such nothing will ever supersede it as we will simply alter our former believes in order to keep our mind at the center of the universe.

Now as for machines that can do everything that humans can do (and more), I still consider this a pipe dream, even if we disregard the word intelligence. This is not because I believe the human mind is superior to machine, it is more because we haven’t conceived of how we would do this. Statistical inference has its limits, even with all the computing power in the world, it will never do what specialized machines like our brains have evolved over millions of years to do (well maybe if we allow our machines to freely evolve and procreate and set them loose in a dynamic self sustaining environment; but that wont provide any benefits for us).

I also have my doubts about the utility of AGI. People state that full self driving cars is easier with AGI, but I haven’t seen any proof of this statement (nor for the utility of AGI driven cars for that matter; we’ve had self driving trains for decades without AGI). With traditional machine learning models, there is tons of utility in the data analysis in all sorts of fields (just like there was with factor analysis and linear regression before it), but with AGI I fail to see what we will get from a vague notion of can do everything humans can do, which doesn’t apply to traditional machines that are specifically programmed for that task. In short, I don’t even think there is a market for AGI.

We apply the concept of intelligence to other species, so I don't see why we couldn't apply it to machines. If people want to redefine intelligence so that humans stay on top, then they are wrong to do so. There is undeniably an objective component to it, even if there are many cultural factors at play as well. If you don't like the word "intelligence", replace it with "competence", "productivity", "performance", etc. and the same argument will apply.

You raised two other issues: the possibility of AGI and the utility of AGI.

I think we have good reasons to believe that it's possible. Three possible future scenarios: (1) we get to the point that we can simulate brains to a high degree of accuracy in software, (2) we get to the point that we can build artificial brains out of neurons made of silicon, (3) we get to the point that synthetic biology is advanced enough that we can engineer new kinds of intelligent organisms. These might be science fiction by today's standards, but they show that there is no issue with conceivability. As an aside, the human brain runs on relatively low power consumption and operations per second, so computational power isn't a bottleneck. We have enough compute, we just don't have the right algorithms yet.

As for the utility of AGI, I strongly disagree that there is no market for it. By definition, an AGI can do anything that a human can do. There is already a job market for humans. As soon as AGI can compete with humans in domains like management, engineering, scientific research, education, etc. I don't see why there won't be a huge economic incentive to adopt it.

You hit the nail on the head here (pun intended). Brains (and other adoptive biochemical processes which reinforce behavior) are way more ingenious then to be described as a simple computer. It is not just neurons firing in patterns with synapses which can reinforce certain pathways. No, there is a whole biome in our bodies, from our guts, our muscles, through hormonal systems and yes, nervous system. Our behavior is also influenced by bacteria and other microorganisms that doesn’t even share our DNA (but sometimes actually interacts with our DNA). This complex machinery has evolved over millions of years, and I don’t think there is any way for us to simulate it. Even if we look outwards, our behavior is just as much influenced by each other as it is by our brains. So for a successful simulation you’d need to “raise” your machine in a similar manner, allowing interactions with a whole community (or more) that treats it as equal. Our brains are merely a tool which encodes this interaction as learned behavior. Just building a brain will miss all of that and give us a lousy model.

In statistics we learn that: “No models are correct, but some models are useful”. I’ve always taken this to mean that there is not always utility in a 100% accurate model. I think this applies for AGI. If we create a machine that can do everything that a human can, and better, what can we use it for? You claim increased competence, productivity, and performance, but we already have machines that do that. And we humans have found a way to work pretty well with those. We even have fully automatic vehicles that transport millions of people every day (e.g. in Vancouver B.C. and Copenhagen). It doesn’t need to be better then humans in everything, just accelerating and decelerating in the right places, it doesn’t need to know the airspeed velocity of an unladen swallow, it will only need to know if something is blocking the tracks. Traditional statistical inference will make these systems better and cheaper, and they will benefit humanity, there is no need for them to have generalized knowledge beyond their immediate utility, so they will never be given one.

Sometimes we get hung up on comparing AI to humans. I wouldn't want an AI that does math and physics in the same way that humans do. Our brains aren't very good at those things. I'm more interested in the potential for AI to find new, more effective ways of working in these areas.

AGIs by their own general purpose nature can improve other AGIs, possibly including themselves. AGIs can improve other less general purpose AIs.

Either you could ask for the AGIs to improve what you need for you, or the AGIs might do it by themselves.

Humans may be like robots but their programing is not statistical pattern matching or Boolean logic. Rather it is the intelligence selected for by evolution to achieve the goals of an individual organism.

How does this no make sense?

Intelligence does not just happen absent goal directed evolution.

As far as I can tell, your points don't apply to what I said. I'll quote myself:

>The idea that machines couldn't do everything a human can do (and more) doesn't make any sense to me.

In other words, it doesn't make sense to me when I hear impossibility arguments about AGI that appeal to something special about humans. I'm not making a claim about software and hardware in 2023. I'm saying in principle, there is nothing stopping us from building artificial systems that are just as complex and capable as evolved biological organisms, other than our lack of knowledge of how to build such systems.

>Software does not because it has not evolved

Software does evolve. To get evolution you need three things: replication, mutation, and selection pressure. We replicate, mutate, and select software all the time. I don't think you need a large scale simulation to get evolutionary dynamics.

>Humans may be like robots but their programing is not statistical pattern matching or Boolean logic

That might be true, but I'm also skeptical. Part of the problem is we don't really understand intelligence very well. If we really understood what makes humans capable of intelligent behavior, we would be able to replicate it in artificial systems. I don't see a fundamental barrier other than having the right algorithms.

And what makes you so sure the opposite won't happen? E.g. GAI will be achieved earlier by excluding some aspect of human intelligence that some people deem essential. The whole "problem" is that we can't agree on what constitutes intelligence. I predict that it won't be long before AI systems can score 200 on standard IQ tests.

I don't know of a "standard" iq test that goes past 160 or so.

ChatGPT is easily circumvented to violate all its own guardrails with a few instructions.

Knowledge without understanding.

It clearly doesn't understand it's own rules well enough to realize when it is asked to violate the rule "don't do C" that being asked to do A->B->C is the same as being asked to do C.

More or less hit the nail on the head :)

Driverless cars don't need to work perfectly all the time. They just need to injure fewer people than human drivers.