Re:Natural language benchmarks don’t measure AI models’ gene...
by tsimionescu on Monday August 24, 2020 @03:51AM
(#24258351)
We obviously have a working model of the world that we draw facts from, that is partly built-in (trained by evolutionary processes) and partly refined with personal learning.
By contrast, GPT-3 only has a knowledge of text. To the extent you could say that it thinks, it thinks in terms of textual symbols that it has seen before - that is its world.
It is theoretically possible that if we found a way to encode experiences of the world of the world in a dataset and used the same training mechanisms and models that we used for GPT-3, we would get an AI with a decent model of the world. But it is magical thinking to imagine that we can get there by training a model on raw text.
My thinking is that we humans have some ability to do things by rote until we have a good statistical understanding, at which point we use the statistical understanding mostly, and fall back to rote where we have to be careful. But, mostly, we're stats machines too.