<rant>That's where graph neural nets come into place. They can learn relations, scaling to a large number of objects. A traditional approach would have to learn all possible combinations, hitting the combinatorial explosion. Graph neural nets can solve problems such as shortest path, sorting and dynamic programming. I think in the future if we are to get closer to human level we need graphs as the intermediate representation. Graphs could represent the objects in an image/phrase and their relations, then answer about the attributes of an object, the relation between two objects or classify the graph itself. All simulators are evolving graphs as well, and code/automata could be represented and executed as a graph. The transformer could be considered an implicit graph where the adjacency matrix is computed from the nodes at each iteration. The closest to AGI in my view would be model based RL implemented with graphs.</>
But what bothers me is that the human animal is trying to create an artificial human mind, the most amazing piece of meat we have laying around.
If we could create the reasoning of a dog first, to then upscale to more complex logic, I'd be more confident research is going places (as in, we have biological evidence building blocks to understand first).
But it's true we don't precisely know how to "find and represent the better theory", if it's at all possible (most likely not) and what the exact trade-off is.
Also note that my argument doesn't necessarily rely on computational difficulty of SAT - even if we limit to some "simple" subset of SAT instances, we might see that the pattern recognizer is unable to learn these problems correctly.
The author is right that what we do for NLP is not enough for reasoning. But it does not mean that we are too far from it.
Even in your variation of the room experiment, if the person in it could reply something along the lines of "here is what I think is a part of solution", which you simply had to feed back to eventually get the whole thing, would you claim it is a weak abstraction?
To see that better, consider what I call "simplified Chinese room". It's a variation on a traditional Chinese room, where inside the room, there is only a pattern recognizer, which basically will match the input to arbitrarily many inputs (but not all possible) it learned before and chooses the output for the best match.
Now imagine I want to train this "simplified Chinese room" on solving satisfiability problem. Because in that problem, an arbitrarily small change in the input (introducing a contradictory clause) can completely change the output. It is therefore impossible, I believe, to learn the concept of satisfiability by just using pattern recognition (storing and comparing, according to some metric, previously seen inputs and corresponding correct outputs). Instead, you need to build a mental model which is internally self-consistent with these example pairs.