I think we need to start by asking a better question and not try to simplify too much. We also need to accept that some answers are complex and not everything can be reduced to a soundbite to be used to end internet discussions.
Let’s take a different question, like “what’s missing from a worm for it to be able to fly”. We might be drawn to the simple answer of “wings” but that isn’t quite right—ostriches and penguins have wings and they don’t fly, so obviously there are other variables at play.
How about “what’s missing from a spec of dust for it to be intelligent”. Well, there isn’t one thing missing and there’s no simple thing we can just add to make a spec of dust intelligent and sentient, its very nature needs to be radically different.
What test would you propose where an LLM (or an RL agent containing an LLM) would reliably fail, but where an average human would reliably succeed?
I’ll say it again:
> I think we need to start by asking a better question and not try to simplify too much. We also need to accept that some answers are complex and not everything can be reduced to a soundbite to be used to end internet discussions.
You couldn’t even do it in your counter-example:
> (…) testable characteristic (at least within certain bounds (…)
Even with a much simplified use case, you still had to add the caveat. That should tell you this is much more complex than arguing with strangers on the internet in search of a “gotcha” moment.
Can you, in a short HN moment, propose a good test to (say) prove or disprove the existence of God, depending on your belief? Of course not, because by the very nature of the definitions we have for God, I could always add a counterpoint to make your experiment fail. That would allow me to ridicule and dismiss your position, whatever it is, because it would be framed within unreasonable constraints. I’m not interested in such dishonest conversations, I don’t see winning arguments in itself as a laudable or interesting goal.
You are completely incorrect. You're falling in the same trap that most humans fall into. That is you're completely incapable of seeing intelligence at different scales.
Cells have intelligence. Organs have intelligence. Bodies outside the brain have intelligence. Hell, many scientists accept that things like proteins likely have intelligence as they can adapt in their environment, and many more are making claims that algorithms have intelligence.
You, as of so far have given no explanatory evidence of where intelligence emerges from, only "I'll know it when I see it". The actual definition of intelligence doesn't work this way. Any, and I mean any neural network is capable of narrow intelligence. Going lower into algorithmic intelligence, the applications and CPU on your computer are intelligent in some measures.
Go outside of your extremely narrow definition of whatever you think intelligence is and learn more about it. You could start studying now and it will take the rest of your life learning more to grasp how far the scales of, the simplicity, and the complexity of intelligence actually go.
Do you treat everyone who doesn’t completely subscribe to your views as being incapable of understanding them? Disagreement with a view does not equal inability to understand it.
> You're falling in the same trap that most humans fall into. (…) Cells have intelligence. (…)
Ironically, it is you who is falling into the trap of letting the technically correct stand in the way of the useful. A definition of intelligence which is too broad is worthless to the practical matters of the discussion. I was replying within a context, that’s what a conversation is. If there are scales, might you not understand the topic of LLM intelligence at hand is concerned with a particular level?
For sure we can have a conversation within a separate framework, but that is a different conversation. Don’t just come in midway, completely change the parameters, then strut around like a peacock of righteousness who is lord and master of universal truths.
> many scientists accept (…) and many more are making claims
This is classic weasel wording¹. You can find “many scientists” to say just about anything. “Many scientists” claim vaccines cause autism and that human actions are not exacerbating climate change.
> You, as of so far have given no explanatory evidence of where intelligence emerges from, only "I'll know it when I see it".
Again, ironic. You completely miss the context, criticise a lot, and repeatedly do the very thing you’re accusing others of.
> Go outside of your extremely narrow definition of whatever you think intelligence is and learn more about it.
I strongly recommend you go outside your extremely narrow definition of other people and learn some humility. To put it bluntly, you came of as a major prick². That is no way to change the mind of others, it only gets them on the defensive. Are you interested in changing opinions or winning arguments? If the former, you did a terrible job. If the latter, then discussing anything with you is devoid of value.
You seem like a smart person, I have no doubt you can do better.
As a free quick tip, replace your first paragraph (the three sentences) with “Intelligence manifests at different scales”. Way shorter, says the same thing, doesn’t make assumptions, and leaves the other person open to a conversation.
¹ https://en.wikipedia.org/wiki/Weasel_word
² Which, crucially, does not mean you are one. We all have moments when we aren’t at our best and we can come of differently from what we intended.
Humans aren’t special. Other animals are intelligent and interesting too. A machine could be intelligent. LLMs aren’t.
In other words, believing in human exceptionalism is not a prerequisite to understand the current crop of AI is not the end all be all of its hype. It is supremely common that AI proponents do not understand that, however. Like hardcore cryptocurrency fans who believe anyone who doesn’t like them is “just jealous they didn’t make bank”, too many hardcore AI proponents believe anyone who doesn’t think LLMs are intelligent is jealous of humans no longer being unique, or afraid for their jobs, or whatever. In both cases it’s obvious that what those proponents lack is empathy, the ability to understand not everyone has the same selfish thoughts they do.