pizza234
This is why AI companies use research in a deterministic and rigorous field (mathematics) as a benchmark - it creates the impression that humanity will achieve comparable progress in other fields.
I'm not implying that research progress will be necessarily very slow, but that it's severely misrepresented for sure.
> Today, recursive self-improvement is often operationalized as a model writing code that launches training jobs. An LLM that can inspect, debug, and improve its own weights, or those of a newer model, offers a richer, more native path to it.
No, Astra is still smarter (https://artificialanalysis.ai/models/?models=gpt-6-astra%2Cg...).
Excluding the modern equivalent of a script kiddie, even a "budget" attacker will have for sure invested enough to run a serious model (GLM 5.3 in this case, which requires hundreds of GB of RAM).
On the other hand - the defense - there is a large spectrum, and that's where the problem lies. There are surely companies that can defend themselves (either via cloud solution with granted cyber capabilities, or on local powerful machines), but there is a large amount of people that don't even know how to run a local LLM, let alone compile a program like Firefox.
It'd be simpler in the post to contrast, say, an "electronic" alien driven by AI, versus a syntethic machine, made by humans, with the same (hypothetical) AI; we'd istinctually recognize the former as intelligent/living, but not the latter.
The post is essentially retelling the carbon chauvinism concept.
So much for all the people stating (foolishly) that the HuggingFace incident was not a big deal because OpenAI's containment was a "wet paper bag". This misses the point: AIs will get better and better, and containment will be harder and harder.
It's 100% sure that at some point, AIs will attempt to cross airgaps.
The same applies to LLMs: we know how they produce language, but the underlying processes that enable it may themselves constitute a form of intelligence.
LLMs may learn from correlations and consistency, but that's separate from developing internal representations of what they believe is true, which is precisely what the paper provides evidence for.
"Believe" is of course not in the strictly human sense.
This level of understanding of LLMs may have been true a few years ago, but we're past far that. LLMs have solved a Millennium Prize problem, just in case.
Having emergent properties implies that an LLM has capabilities beyond the underlying mechanics (stochastic and the like), therefore to recognize emergent properties and at the same time to state that they are just regression machines is self-contradictory.
This is for example an interesting research, that investigates the existince of an internal state of truthfulness: https://aclanthology.org/2023.findings-emnlp.68.pdf. There's a lot of research on the higher level properties of LLMs.
See here: https://aclanthology.org/2023.findings-emnlp.68/ ("The Internal State of an LLM Knows When It’s Lying").
With the advent of agents, easy or hard doesn't matter much. One in thousands of agents will likely find a way, and that's what matters.
The related public challenge (https://github.com/KillaBoi/StealMyBitcoin-PoC) has been hacked after around 19 hours. It's up to debate whether it's really representative (at first it wasn't for sure, but it's been corrected), but shows that there has been an exploitable RCE pathway.
Well, anthropomorhizing is actually a fitting way of thinking AIs: they lie like humans do.
Problem is: there is a limited amount of apps that are genuinely fun and educative. If Amazon wanted really to do something truly educative for children, they'd build a whole market for it, the marketese I see in the article doesn't support that.
Regarding my kids: high priority, wide/curated choice (not in terms of restriction, but in terms of actively finding things they may like), accompanied learning, and, crucially, starting from a very young age. It's pretty much standard parenting, that is, one gets out what one puts in.
By the time children start to pressure for this type of devices (say, 10/12), they will have learned to enjoy books and more creative activities.
There are several reasons. The most important one is that children (ab)using highly stimulating devices don't learn to be absorbed by or find reward in less stimulating ones (most importantly, books). It's exactly the same risk as giving addictive drugs to adults.
This is not theory. Just look at children whose parents give them mobile phones/tablets as pacifiers - they become easily restless.
The intellectual argument is also important. Gamified learning is nice on paper, but one can imagine what children are actually going to play.
Related to another post: the difference between a child and an adult who plays videogames is that the latter has (hopefully!) learned "slow" skills, so they can choose pastimes like this deliberately.
Having said that, I'm for forbidding children from using this type of thing, but it's a very dangerous slippery slope, and there's lots to work on before that step.
Interestingly, according to the parent's logic, there is a specific argument for that (took me a while to get the argument, and I think its not sound).
The reasoning is that, according to reliabilist responses to Gettier's problem, knowledge requires a reliable belief-forming process. But if we assume that LLMs operate probabilistically, they can't fulfill this, because they're "guessing".
The refutation to this argument is the usual one: levels of abstraction. LLM reasoning, like human intellect, can exhibit structured/logical behavior at a higher level of abstraction, even if the underlying processes are stochastic.
Therefore, if one bases the reasoning on the fact that both are stochastic at the lower level, then both can't produce knowledge, but if one bases it on the higher level, they both can.
(Having said that, I don't imply that AIs can reason like humans, but just that the criticism is not sound)
This is false and misinformed.
There are multiple analysis on the HuggingFace incident, and it blows my mind how people refuses to read it. In particular, the METR institute (a 3rd party) has published a very detailed analysis; and they're not the only ones.
Sad thing is that information like this is one Google away, but posters who make such statements are so lazy that they don't even bother.
Pretty much like audio tapes and VHS... floppies :) That was indeed the time of boot viruses!
Later, in the 90s, there were also warez CDs and DVDs - see Twilight CDs :)
The incidents are actually considerably more complex than how they appear to those who just read headlines.
In the HuggingFace accident for example, the sandboxes were not providing internet, and the sandbox hosts did not have internet connection either. Internet was reached by hacking an internal service.
In the DNS accident the situation was similar. Internet connection, including public DNS, was not accessible. The agents ended up exploiting, in a similar fashion, the internal DNS resolver.
It's (very) appropriate to push for security, but equating these accidents to putting gasoline in a microwave is stupid and ill-informed. And dangerous, because it minimizes the capabilities of frontier AIs.
> it’s also important that they learn how to browse, search for information, and work with web applications
Also, keep in mind that isolating complex systems is not simple. Agents need to be in sandboxes, but for example, the sandboxes themselves may need to be kept up to date, or may need to be monitored, or the sandboxes may need to updated, or they may be connected to specific services etc. Not saying that it's impossible, but it's complex.
In the HuggingFace incident for example, the hosts themselves were not connected to the internet. But a service they were using was, indirectly, and that's what the agents attacked.
> It's what the military does, and they use color-coded cables to avoid accidental interconnections between classified and unclassified networks.
Requirements are very likely different. I presume that handling thousands of agents is enormously more complex than secret documents or "static" applications.
In the OpenAI case, agents even hacked the internal OpenAI network. Isolating networks (which was probaly done to an extent at OpenAI) can still result in one network being "infected", and that is... interesting, to say the least.
There's plenty of posters on Hacker news denying or minimizing the powers of AI, in one way or another: pointless arguing on the meaning of "thought"/"rogue", characterizing agents as monkeys and/or defenses as "wet paper bags" in cases of successful hacks, and so on.
Basically, it hurts to admit that we've created something that can/will become more powerful than us, so some will deny it, even in blatant cases.
In my personal expierience, LLMs (as of 2026) lack vision/creativity - they're (so) good at implementing bounded/set ideas, but they don't see the big picture/think out of the box.
Ha. Gettier actually argued that justified true belief is insufficient for knowledge, not that machines can't possess it. And JTB doesn't inherently exclude machines.
Interestingly, research suggests LLMs encode truthfulness in their internal representations: https://aclanthology.org/2023.findings-emnlp.68/
AIs/LLMs don't work like this. You don't feed an input, that gets stored and then reproduced, for two reasons:
1. as for humans, inputs are processed and distributed across a very large network of information
2. AIs may have some level of belief (unless on applies the usual circular logic that they can't, because property X is exclusively human).
There's a very interesting research: https://aclanthology.org/2023.findings-emnlp.68/ in which researchers inferred the truthfulness of statements by examining a internal activations. AIs, in short, appeared to have an internal representation of truthfulness.
> it does not know
Define "knowledge" in a non-circular way.
There's a certain type of knowledge that machines definitely can't have - experiential - but conceptual knowledge doesn't inherently require subjective experience or a biological substrate.
> the only thing it does is reproduce patterns more or less abstract in function of the pressure you put on certain representation during training time.
As for humans; just replace "training" with "education" and the likelihood of a certain output with the likelihood of a certain thought or response.
Amusingly (in this context), much of adult behavior is shaped by patterns reinforced during childhood.
I think it's not a big issue for people who clean up after themselves daily, but on the other hand, sparing those last 10 minutes on the checkout morning... ;)