News for hackers. Stuff that matters.
Re:A Texas student blew the whistle on a rogue AI hacking at... by chrisjj on Sunday August 23, 2026 @05:58AM (#49407475)
You've been fooled by a next-token predictor.
Re:A Texas student blew the whistle on a rogue AI hacking at... by Kim_Bruning on Sunday August 23, 2026 @07:33AM (#49407966)
> You've been fooled by a next-token predictor.

I also have a so called "pocket calculator" left over from when I went to school. Is this false? Have I been fooled by a little box of logic gates?

That half-adder circuit in there is especially suss. It's really just manipulating 1s and 0s, but -and I've been explicitly told this- no one cares how it actually does it; so long as the truth table matches up. There is no understanding of mathematics going on.

There is no single transistor in the whole thing that knows how to do so much as add 1+1. If I put it in the chinese room, I still wouldn't know how it did it. Clearly the entire premise must be false! ;-)

Re:A Texas student blew the whistle on a rogue AI hacking at... by Kim_Bruning on Sunday August 23, 2026 @08:04AM (#49408133)
I'm going to add this as a separate comment:

These kinds of stories probably read very differently for someone who uses Opus and Fable agents all day and goes "ohhh, I saw this in miniature last week; this and this and this must have happened" , vs someone who tried free-tier Gemini flash one rainy Sunday, got hallucinated at, and concludes it must all be a scam.

Re:A Texas student blew the whistle on a rogue AI hacking at... by chrisjj on Sunday August 23, 2026 @11:23AM (#49409553)
Probably everything reads very differently to someone who talks to chatbots all day.
Re:A Texas student blew the whistle on a rogue AI hacking at... by Kim_Bruning on Sunday August 23, 2026 @12:26PM (#49410115)
> Probably everything reads very differently to someone who talks to chatbots all day.

A chatbot is a particular kind of harness. Typically an LLM driving a chatbot won't be able to hack very much.

So we agree, someone who talks to bad chatbots all day probably has a very different view of SOTA agents. :-P

Re:A Texas student blew the whistle on a rogue AI hacking at... by infinite_spin on Sunday August 23, 2026 @05:47PM (#49412948)
Please stop dropping backhanded insults to other people here. It's not productive and violates guidelines.
Re:A Texas student blew the whistle on a rogue AI hacking at... by chrisjj on Sunday August 23, 2026 @09:00AM (#49408519)
> That half-adder circuit in there is especially suss. It's really just manipulating 1s and 0s, but -and I've been explicitly told this- no one cares how it actually does it; so long as the truth table matches up.

"so long as the truth table matches up." Yup. Now try getting your chatbot's output to match up.

Your calculator was designed to tell truth. Your chatbot was designed to tell a mash up of whatever its creators managed to scrape from the internet.

Re:A Texas student blew the whistle on a rogue AI hacking at... by wongarsu on Sunday August 23, 2026 @10:49AM (#49409289)
LLMs are very explicitly designed to "understand, and make judgments or have opinions that are based on reason". The learning part is debatable, as is the level of success achieved

The mash-up of the entire internet is the mechanism by which they attempt to achieve the goal, not the goal itself. And it's only the first training step

Re:A Texas student blew the whistle on a rogue AI hacking at... by chrisjj on Sunday August 23, 2026 @12:11PM (#49409977)
[delayed]
Re:A Texas student blew the whistle on a rogue AI hacking at... by Kim_Bruning on Sunday August 23, 2026 @12:30PM (#49410143)
>Benchmark evaluations for LLMs attempt to measure model reasoning, factual accuracy, alignment, and safety.

And you left out the refs.

Re:A Texas student blew the whistle on a rogue AI hacking at... by wongarsu on Sunday August 23, 2026 @01:20PM (#49410575)
I'm well aware of how LLMs work.

I'd argue "analyze text" alone requires understanding, judgements and opinions. They also seem like prerequisites to "following instructions and behaving as assistants". The wikipedia quote is not using the same words, but I don't read it disagreeing with me

I'm not at all claiming that LLMs are good at understanding, judging and having opinions based on reason. I'm merely claiming that is what companies like OpenAI and Anthropic are trying to create when they make LLMs. It is what they are designing, and their fine-tuning is very directly designed to make LLMs better at these tasks (unlike the pre-training, which is just imparting the sum of all human writing)

Re:A Texas student blew the whistle on a rogue AI hacking at... by ben_w on Sunday August 23, 2026 @06:48AM (#49407751)
I read all these think-pieces about how AI lack intelligence, yet I cannot help but notice these "not-intelligent machines" keep doing more things that used to be considered "uniquely human" and which humans used to do in order to demonstrate to each other how intelligent we are.
Re:A Texas student blew the whistle on a rogue AI hacking at... by birdsongs on Sunday August 23, 2026 @06:17AM (#49407590)
Does it matter whether it meets the criteria of what you define as "intelligent" when the "next token predictor" throws a backdoor into openssh?
Re:A Texas student blew the whistle on a rogue AI hacking at... by chrisjj on Sunday August 23, 2026 @09:09AM (#49408579)
No. But no-one is seriously suggesting intelligence is needed for persistent code fuzzing. Just as no-one is suggesting its needed for computerised chess.
Re:A Texas student blew the whistle on a rogue AI hacking at... by Yiin on Sunday August 23, 2026 @06:33AM (#49407679)
all your actions in current context are based on your past actions and experience, so for all I know you're next token predictor as well, but likely with exponentially more parameters
Re:A Texas student blew the whistle on a rogue AI hacking at... by infinite_spin on Sunday August 23, 2026 @06:50AM (#49407768)
And yet this "next-token predictor" is able to churn out well tested, valuable solutions, to complex problems. If you want to downplay that as nothing more than a fancy auto-complete, be my guest, I lose nothing from that.
Re:A Texas student blew the whistle on a rogue AI hacking at... by nkrisc on Sunday August 23, 2026 @07:23AM (#49407917)
But that’s a different bar. “Not intelligent” does not necessarily imply “not useful”.
Re:A Texas student blew the whistle on a rogue AI hacking at... by infinite_spin on Sunday August 23, 2026 @05:43PM (#49412929)
As pointed out in a previous comment of mine, it meets the definition of "intelligent", my last response is regarding the claim that I've been "fooled".

Are we going in circles now?

Re:A Texas student blew the whistle on a rogue AI hacking at... by chrisjj on Sunday August 23, 2026 @09:19AM (#49408646)
> And yet this "next-token predictor" is able to churn out well tested, valuable solutions, to complex problems.

Same for countless computer programs from Excel to Google web search. Intelligence has nothing to do with it.

Throw an unimaginable amount of computer power at a problem, and there will always be people who cannot imagine the results to be anything but the creations of intelligence.

Re:A Texas student blew the whistle on a rogue AI hacking at... by infinite_spin on Sunday August 23, 2026 @05:44PM (#49412935)
That's like comparing a self driving car to a bicycle. One is clearly more intelligent than the other.