And I don’t think the problems computers have proven themselves useful in solving have been what most people expected. Chess, Go, facial recognition, Jeopardy, image classification (hot dog or not), captchas (clearly, since they’re designed specifically to resist computer solutions), etc. seem to me to be things that, before computers proved to be decent at, would have been widely considered to require intelligence on the level of humans.
But there aren't any formally-specified rules for driving cars, and this isn't obvious.
Nope, despite a myriad of road codes, the actual traffic doesn't follow a set of formalized rules: a chess rook can't just decide that it will start disintegrating all of a sudden, as opposed to a vehicle. You could probably approximate the ruleset if you made it self-modifying...which will then demolish your second point about near-exhaustively searching the state space - good luck doing that before the heat death of the universe, as you're essentially simulating the whole environment. Oh look, there's also weather. How's that exhaustively searchable? Asking for the Nobel Prize committee.
For the sake of discussion, let's say that a miracle happens and you managed to do all that - but sorry, it's useless again, the few seconds have already elapsed and you need to do it again. And again. And again, ad infinitum.
Now, I could envision "by our current technology, we can't yet, but we're hoping for a miracle in this specific spot" - but "assuming a massive miracle happens every few seconds, for each vehicle" is completely removed from reality: why not have teleports, if we're in magical wish-granting land already?
For highway driving Waymo had 6 disengagements in 2017, street: 57.
Total driven: 352000 miles
1 disengagement for "a recklessly behaving road user" 5 for "incorrect behavior prediction of other traffic participants"
Seems like predicting other people is almost perfect, the others were more internal problems.
Driving a car when a weird thing happens isn't that complicated: You stop, braking at the minimum amount required to do so safely, to avoid cars behind you hitting you
(In other words, yes, it might be eventually possible to have self-driving vehicles, but pretending that the search space is bounded, or even near-exhaustively searchable a la chess - that's just pure technobabble)
Anyway, self driving cars will only get better, and the more of them there are the better they will be, because no humans around doing weird things who don't talk over the SDC network to explain where they're going
Rather, I suspect, tasks which computers start outperforming humans in we reanalyse as "completely procedural". Nobody called chess procedural in the middle of last century.
Historically AI chess (e.g. "Deep Blue" or Stockfish) is played by machines using one heuristic to estimate how "good" positions are without truly knowing, not so dissimilar from how humans evaluate a chess position. and then another heuristic to try out moves to get to further positions. The machine considers possible plays and how they affect the heuristic "value" of the board, preferring those with more value. Human Chess AI authors design the two heuristics used, though they often aren't very good at actually playing chess because it's a different skill.
Google's AlphaZero AI plays chess differently again, it had no preconceptions of how to play Chess, instead it learned through self-play - it knows the rules of the game but began with no idea what's a good or bad move, it adapted its own heuristics based on how well they'd won or lost. It actually recapitulated most of human chess theory history over its incubation period of thousands of games, discovering ideas like the Sicilian Defence for itself, new attacks would at first see overwhelming success, and then, playing versions of itself that had seen these attacks, they'd be defended more effectively.
Alpha Zero plays a radically "more human" style of chess than most modern human Chess grandmasters, huge multi-move strategies in which pieces are sacrificed to take positional advantage. It looks like something humans were doing last century - except Alpha Zero does it much better than they ever did.
The problem is that you lack an evaluation function. Let's consider two of those 100 possible moves. Your rook could take this opposing pawn, or, your own pawn could move forward one space. Which is better? Why? Neither of them immediately wins the game, but we must pick something. In a smaller, tighter game, like Tic-Tac-Toe we could crank our exhaustive search until we discover that this opening move leads to a possible win... but the search space in Chess is categorically too enormous for that.
Both Google's Alpha Zero and simple human play encourages the belief that a good evaluation heuristic is essential. The evaluation heuristic looks at a board position and it doesn't recommend a move it says something like "I rate this position 0.418" where 1.0 is "I'll definitely win on my turn" and -1.0 is "My opponent wins on their turn". Google's engine contemplates relatively few possible moves (for a computer) but the results are striking because it's looking at _good_ moves more of the time rather than wasting a lot of time thinking about moves that are a bad idea.
This seems obvious, but, well, learn chess and see for yourself.
Here's an example of a simple chess engine that is good enough to beat amateur human players at least some of the time:
Did they though? Think about things that computers can do better than people: they're mainly things that we completely predicted computers would be better at (arithmetic, precision manufacturing, drafting, telecommunications routing). Beyond that, you're left with things computers are only better at dependent on priority, the canonical example being service jobs where economics trump's QoS; computers are much worse than a cashier, but comparatively cheaper by a margin that makes the quality compromise worth it.
The only possible exception I could think of that's come up in recent discourse is diagnosing patients, but even that, while encroaching on a role that has traditionally been revered as a career, is still something that seems at least on the surface to be quite predictable given the nature of what's required to make diagnoses (simultaneous access to a trove of data and knowledge).
Beyond the above, I think it's pretty reasonable that there's a broad range of things computers will not be better than humans at for a very very long time, if ever.