Re:The "confident idiot" problem: Why AI needs hard rules, n...
by rockskon on Monday December 08, 2025 @12:26PM
(#46195018)
Analogies of LLMs to humans obfuscates the problem. LLMs aren't like humans of any sort in any context. They're chat bots. They do not "think" like humans and applying human-like logic to them does not work.
[ Parent
| on HN ]
Re:The "confident idiot" problem: Why AI needs hard rules, n...
by not2b on Monday December 08, 2025 @01:23PM
(#46195768)
You're right, mostly, but the fact remains that the behavior we see is produced by training, and the training is driven by companies run by execs who like this kind of sycophancy. So it's certainly a factor. Humans are producing them, humans are deciding when the new model is good enough for release.
Re:The "confident idiot" problem: Why AI needs hard rules, n...
by rockskon on Monday December 08, 2025 @01:27PM
(#46195832)
Do you honestly think an executive wanted a chat bot that confidently lies?
Re:The "confident idiot" problem: Why AI needs hard rules, n...
by jandrese on Monday December 08, 2025 @02:37PM
(#46196612)
Do the lies look really good in a demo when you're pitching it to investors? Are they obscure enough that they aren't going to stand out? If so no problem.
Re:The "confident idiot" problem: Why AI needs hard rules, n...
by jacquesm on Monday December 08, 2025 @01:46PM
(#46196057)
Given the matrix 'competent/incompetent' / 'sycophant/critic' I would not take it as read that the 'incompetent/sycophant' quadrant would have no adherents, and I would not be surprised if it was the dominant one.
Re:The "confident idiot" problem: Why AI needs hard rules, n...
by dontlikeyoueith on Monday December 08, 2025 @01:41PM
(#46196000)
In practice, yes, though they wouldn't think of it that way because that's the kind of people they surround themselves with, so it's what they think human interaction is actually like.
Re:The "confident idiot" problem: Why AI needs hard rules, n...
by rockskon on Monday December 08, 2025 @01:57PM
(#46196187)
"I want a chat bot that's just as reliable at Steve! Sure he doesn't get it right all the time and he cost us the Black+Decker contract, but he's so confident!"
You're right! This is exactly what an executive wants to base the future of their business off of!
Re:The "confident idiot" problem: Why AI needs hard rules, n...
by Retric on Monday December 08, 2025 @05:58PM
(#46198839)
You say that like it’s untrue, but they measurably prefer a lying but confident salesman over one who doesn’t act with that kind of confidence.
This is very slightly more rational than it seems because repeating or acting on a lie gives you cover.
Re:The "confident idiot" problem: Why AI needs hard rules, n...
by dontlikeyoueith on Monday December 08, 2025 @08:39PM
(#46200303)
Yes, that is in fact their revealed preference.
Did you have a point?
Re:The "confident idiot" problem: Why AI needs hard rules, n...
by rockskon on Tuesday December 09, 2025 @12:50AM
(#46201683)
You use unfalsifiable logic. And you seem to argue that, given the choice, CEOs would prefer not to maximize revenue in favor of... what, affection for an imaginary intern?
Re:The "confident idiot" problem: Why AI needs hard rules, n...
by dontlikeyoueith on Tuesday December 09, 2025 @02:36PM
(#46209512)
Cute straw man.
You must be a CEO.
I'm not arguing anything. I'm observing reality. You're the one who is desperate to rationalize it.
Re:The "confident idiot" problem: Why AI needs hard rules, n...
by rockskon on Tuesday December 09, 2025 @02:58PM
(#46209818)
You are declaring your imagined logic as fact. Since I do not agree with the basis upon which you pin your argument on, there is no further point in discussion.
Re:The "confident idiot" problem: Why AI needs hard rules, n...
by dontlikeyoueith on Tuesday December 09, 2025 @08:51PM
(#46213129)
You're hallucinating things I did not say.
Re:The "confident idiot" problem: Why AI needs hard rules, n...
by ryandrake on Monday December 08, 2025 @03:23PM
(#46197157)
They may say they don't want to be lied to, but the incentives they put in place often inevitably result in them being surrounded by lying yes-men. We've all worked for someone where we were warned to never give them bad news, or you're done for. So everyone just lies to them and tells them everything is on track. The Emperor's New Clothes[1].
1: https://en.wikipedia.org/wiki/The_Emperor%27s_New_Clothes
Re:The "confident idiot" problem: Why AI needs hard rules, n...
by not2b on Monday December 08, 2025 @01:40PM
(#46195991)
No, but they like the sycophancy.
Re:The "confident idiot" problem: Why AI needs hard rules, n...
by mrguyorama on Monday December 08, 2025 @01:55PM
(#46196171)
People with immense wealth, connections, influence, and power demonstrably struggle to not surround themselves with people who only say what the powerful person already wants to hear regardless of reality.
Putin didn't think Russia could take Ukraine in 3 days with literal celebration by the populace because he only works with honest folks for example.
Rich people get disconnected from reality because people who insist on speaking truth and reality around them tend to stop getting invited to the influence peddling sessions.
Re:The "confident idiot" problem: Why AI needs hard rules, n...
by Retric on Monday December 08, 2025 @01:25PM
(#46195803)
It’s not about thinking, it’s about what they are trained to do. You could train a LLM to always respond to every prompt by repeating the prompt in Spanish, but that’s not the desired behavior.