News for hackers. Stuff that matters.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by admiralrohan on Wednesday March 19, 2025 @05:56PM (#43417705)
Hallucinating is fine but overconfidence is the problem. But I heard it's not an easy problem to solve.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by jacksnipe on Wednesday March 19, 2025 @07:14PM (#43418232)
Unfortunately, in the system most of us work in today, I think overconfidence is an intelligent behavior.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by onemoresoop on Wednesday March 19, 2025 @07:29PM (#43418348)
I find it extremely dumb to see overconfident people that really have nothing special about them or are even incompetent. These people are not contributing positively to the system, quite on the contrary.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by xwolfi on Wednesday March 19, 2025 @08:08PM (#43418629)
But we don't work for the system, fundamentally, we work for ourselves, and the system incentivizes us to work for it by aligning our constraints: if you work that direction, you'll get that reward.

Overconfident people ofc do not contribute positively to the system, but they skew the system reward's calculation towards them: I swear I've done that work in that direction, where's my reward ?

In a sense, they are extremely successful: they managed to do very low effort, get very high reward, help themselves like all of us but at a much better profit margin, by sacrificing a system that, let's be honest, none of us care about really.

Your problem maybe, is that you swallowed the little BS the system fed you while incentivizing you: that the system matters more than yourself, at least at a greater extent than healthy ?

And you see the same thing with AI: these things convince people so deeply of their intelligence that it blew to such proportion that NVidia is now worth trillions. I had a colleague mumbling yesterday that his wife now speaks more with ChatGPT than him. Overconfidence is a positive attribute... for oneself.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by onemoresoop on Thursday March 20, 2025 @02:44AM (#43420346)
Not talking about THE system or any system in particular same way gaming the system doesn’t refer to any system but just cheating in general. And if you like overconfident people good for you, I can’t stand them because they’re hollow with no basis in reality, flawed like everyone, just pumping out their egos with hot air. And your reasoning that overconfidence is a positive attribute doesn’t make much sense to me but we’re entitled to our own opinions.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by butlike on Thursday March 20, 2025 @09:44AM (#43423282)
Overconfident people are conquerors. Conquerors do not contribute positively to a harmonious system, true, but I'm not so sure we can glean the system is supposed to be harmonious.

If one contributes "positively" to the system, everyone's value increases and the solution becomes more homogenized. Once the system is homogenized enough, it becomes vulnerable to adversity from an outside force.

If the system is not harmonious/non-homoginized, the attacker would be drawn to the most powerful point in the system.

Overconfident people aren't evil, they're simply stressing the system to make sure it can handle adversity from an outside force. They're saying: "listen, I'm going to take what you have, and you should be so happy that's all I'm taking."

So I think overconfidence is a positive attribute for the system as well as for the overconfident individual. It's not a positive attribute for the local parties getting run over by the overconfident individual.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by jacksnipe on Thursday March 20, 2025 @04:00PM (#43428169)
Yeah this is what I meant, both in the behavior being intelligent and it being unfortunate that this is the case. It’d be nice if the most self-maximizing behavior were also the best behavior for the global system, but it doesn’t seem that it is.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by tomComb on Wednesday March 19, 2025 @08:20PM (#43418696)
But being like that can get you elected.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by Freedom2 on Wednesday March 19, 2025 @08:55PM (#43418901)
In what country?
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by tbossanova on Wednesday March 19, 2025 @09:13PM (#43419004)
Quite a few :(
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by chromehearts on Thursday March 20, 2025 @05:28AM (#43421166)
All of them
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by sim7c00 on Thursday March 20, 2025 @08:14AM (#43422080)
and getting yourself elected while being underqualified is intelligent? i think its not. its stupid and damaging behavior based in selfish desires. about as far from intelligent you can get.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by rcxdude on Thursday March 20, 2025 @08:43AM (#43422334)
Intelligence is seperate from goals: if you're only interested in gaining power and wealth for yourself, then concern about the rest of the system is only incidental to what you can get for yourself.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by throw4847285 on Thursday March 20, 2025 @09:30AM (#43423059)
That's a fallacy. There are certainly some unqualified elected leaders, but humans living in democratic societies have yet to shake the mental framework we've constructed from centuries without self-rule. We invest way more authority into a single executive than they actually have, and blame them for everything that goes wrong despite the fact that modern democracies are hugely complex systems in which authority is distributed across numerous people. When the government fails to meet people's needs, they lack the capacity to point at a particular Senator or a small party in a ruling coalition and blame them. It's always the executive.

Of course, the result is that people get fed up and decide that the problem has been not that democratic societies are hard to govern by design (they have to reflect the disparate desires of countless people) but that the executive was too weak. They get behind whatever candidate is charismatic enough to convince them that they will govern the way the people already thought the previous executives were governing, just badly. The result is an incompetent tyrant.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by butlike on Thursday March 20, 2025 @09:34AM (#43423112)
Overconfidence allows you to act, increasing survivability. Thinking is a "weak" trait.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by Terr_ on Thursday March 20, 2025 @12:52AM (#43419978)
> overconfidence is the problem.

The problem is a bit deeper than that, because what we perceive as "confidence" is itself also an illusion.

The (real) algorithm takes documents and makes them longer, and some humans configured a document that looks like a conversation between "User" and "AssistantBot", and they also wrote some code to act-out things that look like dialogue for one of the characters. The (real) trait of confidence involves next-token statistics.

In contrast, the character named AssistantBot is "overconfident" in exactly the same sense that a character named Count Dracula is "immortal", "brooding", or "fearful" of garlic, crucifixes, and sunlight. Fictional traits we perceive on fictional characters from reading text.

Yes, we can set up a script where the narrator periodically re-describes AssistantBot as careful and cautious, and that might help a bit with stopping humans from over-trusting the story they are being read. But trying to ensure logical conclusions arise from cautious reasoning is... well, indirect at best, much like trying to make it better at math by narrating "AssistantBot was good at math and diligent at checking the numbers."

> Hallucinating

P.S.: "Hallucinations" and prompt-injection are non-ironic examples of "it's not a bug, it's a feature". There's no minor magic incantation that'll permanently banish them without damaging how it all works.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by plaguuuuuu on Thursday March 20, 2025 @08:11PM (#43430528)
I'd love to know if the conversational training set includes documents where the AI throws its hands up and goes "actually I have no idea". I'm guessing not.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by Terr_ on Thursday March 20, 2025 @08:47PM (#43430743)
There's also the problem of whether the LLM would learn to generate stories where the AssistantBot gives up in cases that match our own logical reasons, versus ones where the AssistantBot gives up because that's simply what AssistantBots in training-stories usually do when the User character uses words of disagreement and disapproval.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by patates on Thursday March 20, 2025 @03:48AM (#43420626)
Hallucinating is a confidence problem, no?

Say, they should be 100% confident that "0.3" follows "0.2 + 0.1 =", but a lot of floating point examples on the internet make them less confident.

On a much more nuanced problem, "0.30000000000000004" may get more and more confidence.

This is what makes them "hallucinate", did I get it wrong? (in other words, am I hallucinating myself? :) )