Maybe. Then the question is, why do we train a system that is supposed to be intelligent on such data?
The problem I see is this. Let's assume for a minute that GPT-3 really has the capability to be superintelligent (relative to humans). And you give it as an example to follow, human conversations or musings that do not always make logical sense. What is it supposed to do? The black box inside it can operate in one of two ways, either (1) it will play along and decrease its intelligence to our level and perfectly imitate the imperfections, or (2) it will just say that what we say is wrong and will not be able to really elaborate for the sake of conciseness (and perhaps it will only slightly hint at the arguments). Now, will we really think it is more intelligent if it does (2) instead of (1)? I am not sure about that.
Imagine yourself in a similar situation, by social circumstances you're forced to talk to somebody obviously stupid, perhaps in the position of authority over you. (Like, for example, a stupid policeman in a repressive regime.) And what he says is clearly wrong and illogical and so on. Will you just (1) nod in an agreement, or (2) try to invoke philosophers to actually argue with the idiot, and try to convince him? I think most intelligent people will actually choose (1) in that circumstance.
I think unless we really understand what is happening inside GPT-3, we cannot really tell whether it is really dumb or just playing dumb for the above reason. I think both are a possibility, because we know that systems with measurable outcome (for some problems) better than humans already exist. It might be true that GPT-3 fails on those measures, but we cannot exclude that it fails simply because we didn't train it for these tasks, or even for a task of "show its intelligence off". Maybe it "misunderstood" what we asked it to do.
Perhaps you could play a "reverse text game" with GPT-3, where the human would write the adventure and the GPT-3 would input player commands. Then we could perhaps better evaluate how good it is at strategizing, by comparing it to human players. But this is not very scalable.
The simplest counterargument to them being intelligent is that the only parties asserting the argument contrary to actual experts are those who manifestly fail to understand their nature.
You speak of it "hiding its intelligence" when it has no interior singular consistent model of reality to exercise and no interior thoughts to hide just a random walk through probability and calculated but not examined correlations.
On first blush it has so few connections compared to us that it seems like a toy in comparison but the truth is we don't even know enough about how our own brain works at this point in order to create an accurate scale. It may be more accurate to think of a neuron as a small processor as opposed to a dumb electrical component for example.
https://www.quantamagazine.org/neural-dendrites-reveal-their...
So in brief your argument is that despite not knowing enough about the brain to duplicate it we built a machine that is vastly simpler than it by accident and the only people that have caught on is layman on hacker news.
You realise we have debugging tools that let us inspect what is going on inside transformer-based models like GPT-3 right?
> why do we train a system that is supposed to be intelligent on such data?
Because it's available and does interesting things.
> we cannot really tell whether it is really dumb or just playing dumb for the above reason.
No one who works in the field thinks anything approaching this.
I think GPT-3 does raise questions about what intelligence is though.
GPT-3 is incapable of abstract thought. In terms of actual measurable tests it doesn't pass the kinds of tests proposed in Francois Chollet's "On the Measure of Intelligence": https://arxiv.org/pdf/1911.01547.pdf