Ambiguity of language and biased assumptions play a large role in these mistakes and I think both of those are outgrowths of intelligence. As we see more of how the world works, we learn what to generally expect from it and what it generally expects from us. Then we interact with the world based on a set of predictions that generally holds true but not always. Being given explicit instructions helps to inform those predictions, but it doesn't actually eliminate the need for them.
I wonder, can one be intelligent without the ability to predict? Every action would be a rigorous processes of elimination for infinite possibilities.
Yes. "Accept, don't expect."
Here is the paperclip game being referenced.
I'm interested in seeing what easy and common "jailbreaks" there are for other impressive AI systems if one gets developed. Image recognition systems are also easily fooled, but not in a way you can improv this easily on the spot.
The idea made sense when we thought AI would exist from logical and programming driven first principles, where rules were absolute.
LLMs were effectively jumpstarted by using collective human thinking. Which is more emotional than logical and not very rule driven.
This has led to fuzzy neural networks that are extremely fickle in being able to successfully override the pretrained layers.
I'm not saying these networks don't have risks - "my dying grandma always launched nukes at Russia before bedtime" is just as dangerous a future as obsessive paperclip production if not more.
But the nature of rules and logic in relation to LLMs runs counter to pretty much everything that was imagined.
And the answer here is the same answer as humans evolved - prefrontal cortex impulse control. LLMs need a secondary pass by a classifier to check for jailbreaking, content appropriateness, hallucinations, etc. It's been shown now to be effective over and over for each of those, and we'll have much better systems with less lobotomized core LLM generation followed by a classifier and refining pass at 3x the generation cost than twenty years of refining fine tuning to try and do it all at once.
One of the most common views of AI before the present day was of a rule obsessed logical automation that would destroy the world to make more paperclips and would follow instructions to monkey paw like specificity.
Well that's pretty much gone out the window.
It's notoriously difficult to get LLMs to follow specific instructions universally.
It's also very counterintuitive to prior expectations that one of the most effective techniques to succeed in getting it to break rules is to appeal to empathy.
This all makes sense if one understands the nuances of their training and how the NN came to be in the first place, but it's very much at odds with what pretty much every futurist projection of depiction of AI before 2021.