I suppose it's a subset of "well defined rules", but it's worth calling out explicitly, I think: Chess also has an objective (and trivially verifiable) win condition.

There are few interesting situations in real life where such a thing exists.

In that regard, AlphaGo is a lot more impressive, not just because it is a vastly more complex problem to find a loss function for playing Go (compared to chess), but also because AlphaGo basically learned to play the game by itself without even having a model of the game rules initially (or so I've heard).

That said, I would still not consider it anything like generalized AI, if only because the set of possible (valid) actions at any point in the game is tiny, while in real-world problems it is basically infinite.

Now I want AlphaGo to teach me. Verbally. Like humans do.

Agreed and thanks for calling it out. It is so much harder to “learn” without obvious win states.

> perfect information game, finite problem space, well defined rules

Chess isn't reality, but the things you list are all mostly things that humans can't deal with either. Take imperfect information - humans can't make decisions using information they don't have any more than machines can. Humans certainly can't deal with the infinite (and their approximations to do so are probably measurably worse than those a computer uses, because a computer can use honest-to-goodness probability formulas).

Operating without rules is not clear cut, but most people do invent a whole heap of funny rules because they can't operate without clear rules either. Humans often literally hate and fear things that look different or don't follow all the funny rules they come up with.

These are the same arguments as are deployed against self driving cars - if a computer doesn't have the information needed to make a decision then neither will a human in the same situation.

The threat, opportunity and potential of AGI is very real. Once technology settles down and stops changing then we'll know that the situation has stabilised. But even as it stands what we have now would easily pass muster as AGI for the 1910s and it is still improving extremely rapidly.

I remember when I was a teenager: a moonless night in the woods and my bicycle light was broken. It was so dark that I literally couldn’t see my hand in front of me. I could drive (slowly) because I knew that part of the street by hard and I knew there is a gravel patch on each side of the road. So I every time I entered gravel from the left I just turned a little more to the right and vice versa. And I knew I entered gravel by ear.

There was also a faint red light from a memorial site candle that I could use as a orientation point.

The thing was that I have never done this before (or after), nor did I think I ever would ever find myself in such a situation. I drove through that very road often at night also often without light because I had shitty broken bicycles, but this night was truly exceptionally dark and the darkest night I ever had seen since.

The question is, what would an AI have done in a similar situation (sensors go dark for some reason)?

Probably have the common sense to stop. It's a trivial test case.

And also an inferior solution.