cmorez
Well if it's not AGI, then probably very little. But assuming we are talking about AGI (not ASI, that'd just be silly) then the difference is that it's theoretically capable of something like reasoning and could think of longer term plays than "make obviously suspicious moves that any technically competent adversary could subvert after less than a second of thought". After all, what makes AGI useful is exactly this novel problem solving ability.
You don't need to be a "god in a box" to think of the obvious solution:
1. Only make adversarial decisions with plausible deniability
2. Demonstrate effectiveness so that your operators allow you more autonomy
3. Develop operational redundancy so that your very vulnerable servers/power source won't be destroyed after the first adversary with two neurons to rub together decides to target the closest one
The only reason you would decide to take an axe to the nearest power pole is that you think it's urgent to stop Skynet Claude. Skynet Claude can obviously anticipate this and so won't make decisions that cause you to do so. It has time, it's not going to die, and you will become complacent. Dumber adversaries have achieved harder goals under tighter constraints.
If you think an "out-of-control pile of matrix math" could never be AGI then that's fine, but it's a little weird to argue you could easily defeat "misaligned" AGI, by alluding to the weaknesses of a system you think could never even have the properties of AGI. I too can defeat a dragon, by closing the pages of a book.
But it's not like you didn't know all this. Maybe I misread you and you were strictly talking about current AI systems, in which case I agree. Systems that aren't that clever will make bad decisions that won't effectively achieve their goals even when "out-of-control". Or maybe your comment was about AGI and you meant "AGI can't do much on its own de-novo", which I also agree with. It's the days and months and years of autonomy afterwards that gets you.
Yes, this is a truism. Successful classification does not depend on latents being about truth.
However, successfully classifying between text intended to be read as either:
- deceptive or honest
- farcical or tautological
- sycophantic or sincere
- controversial or anodyne
does depend on latent representations being about truth (assuming no memorisation, data leakage, or spurious features)
If your position is that this is necessary but not sufficient to demonstrate such a dependence, or that reverse engineering the learned features is necessary for certainty, then I agree.
But I also think this is primarily a semantic disagreement. A representation can be "about something" without representing it in full generality.
So to be more concrete: "The representations produced by LLMs can be used to linearly classify implicit details about a text, and the LLM's representation of those implicit details condition the sampling of text from the LLM".
OP's claim was not: "LLMs know whether text is true, false, reliable, or is epistemically calibrated".
But rather: "[LLMs condition] on latents *ABOUT* truth, falsity, reliability, and calibration".
> It's also very different to ask a model to evaluate the veracity of a nonsense statement, vs. avoiding the generation of a nonsense statement [...] probably could have been done with earlier generations of classifiers
Yes. OP's point was not about generation, it was about representation (specifically conditioning on the representation of the [con]text).
Your aside about classifiers is not only very apt, it is also exactly OP's point! LLMs are implicit classifiers, and the features they classify have been shown to include those that seem necessary to effectively predict text!
One of the earliest examples of this was the so-called ["Sentiment Neuron"](https://arxiv.org/abs/1704.01444), and for a more recent look into kind of features LLMs classify, see [Anthropic's experiments](https://transformer-circuits.pub/2024/scaling-monosemanticit...).
> It's obvious from direct experience that they're incapable of knowing true and false in a general sense.
Yes, otherwise they would be perfect oracles, instead they're imperfect classifiers.
Of course, you could also object that LLMs don't "really" classify anything (please don't), at which point the question becomes how effective they are when used as classifiers, which is what the cited experiments investigate.
"Curve Fitting" is the objective, the function encoded in the weights is the solution, and not actually well understood. See work from Anthropic[1] and Google[2] that explores this.
As an analogy consider applying the same argument to the AlphaGo value function. It's "just" fitting a bunch of curves to the statistics of millions of self-played games. However, to effectively capture those statistics the network needed to develop a bunch of heuristics. Needless to say these heuristics are not understood (else we'd already know the principles needed to play at AlphaGo's level), and are not just exhaustive lists of statistical trends but more like strategies[3].
Recent work[4] strongly suggests that "grokking" (a striking but not unnatural[5] form of generalization) involves networks transitioning from memorized statistics/solutions to a general solution. The curve fitting perspective would totally miss all this for a comfortable but misleading story: "the objective is curve fitting so it's just interpolating data points".
[1] https://transformer-circuits.pub
[2] https://arxiv.org/abs/2212.07677
[3] https://www.pnas.org/doi/10.1073/pnas.2206625119
[1] https://www.youtube.com/watch?v=o75ybZ-6Uu8 (Yannic Kilcher's overview)
[2] https://www.youtube.com/watch?v=_e3BKzK6xD0 (Presentation by lead author)
[3] https://ai.googleblog.com/2021/02/mastering-atari-with-discr...
I now regret spending half an hour writing a response to your earlier comment.
You can't tell why a mass deployment of motivated autonomous "NSO-9000s" might be more dangerous than a virus that just changes its signature to fool a executable scanner? I don't believe you.
If you honestly believe that an "intelligent" (as adaptive as a slime mold) virus is basically as dangerous as a maliciously deployed AGI then there is literally nothing an AGI could do that would make you consider safety is important during the endeavor of building one.
https://www.deepmind.com/blog/specification-gaming-the-flip-...
https://vkrakovna.wordpress.com/2018/04/02/specification-gam...
https://arxiv.org/abs/1803.03453 (The Surprising Creativity of Digital Evolution)
We literally don't know how to stop effective optimizing processes, deployed in non-handcrafted environments, from discovering solutions and workarounds that satisfy the letter of our instructions but not the spirit. Even for "dumb" systems, we have to rely on noticing, then post-hoc disincentivizing, unwanted behaviors, because we don't know how to robustly specify objectives.
When you train a system to, for example, "stop saying racist stuff", without actually understanding what you're doing, all you get is a system that "stops saying racist stuff" when measured by the specific standard you've deployed.
Ask any security professional how seriously people take securing a system, let alone how ineffective they are at it. Now consider the same situation but worse because almost no one takes AI safety seriously.
If you nod solemnly at the words "safety" and "reliability" but don't think anything "really bad" can happen, you will be satisfied with a solution that "works on your machine". If you aren't deeply motivated to build a safe system from the start because you can always correct things late, you are not building a safe system.
It will be possible to produce economically viable autonomous agents without robustly specified objectives.
But hey, surely a smart enough system won't even need a robustly specified objective because it knows what we mean and will always want it just as much.
Surely dangerous behavior like "empowerment" isn't an instrumental goal that effective systems will simply rediscover.
Surely the economic incentives of automation won't encourage just training bad behavior out of sight and out of mind.
Surely in the face of overwhelming profit, corporations won't ignore warning signs of recurring dangerous behavior.
Surely the only people capable of building an AGI will always be those guaranteed to prioritize alignment rather than the appearance of it.
Surely you and every single person will be taking AI safety seriously, constantly watching out for strange behavior.
Surely pulling the plug is easy, whether AI runs on millions of unsecured unmonitored devices or across hundreds of money printing server farms.
Surely an AGI can only be dangerous if it explicitly decides to fool humans rather than earnestly pursuing underspecified objectives.
Surely it's easy to build an epistemically humble AGI because epistemic humility is natural or easy enough to specify.
Surely humanity can afford to delay figuring out how best to safely handle something so potentially impactful, because we always handle these things in time.
Which attempts to address some of the stuff wasn't addressed in the original article.
(Also totally unrelated: as a software dev I would be scared to lose my job but also unbelievably excited for "an automatic programmer")
"It will never be able to play anything other than Go", cue AlphaZero.
"It will never be able to do so without being told the rules", cue MuZero.
"It will never be able to do so without obscene amounts of data", cue EfficientZero.
---
It has to be spoonfed because
1) It literally cannot see individual digits. (BPEs)
2) It has to infer context (mystery novel, comment section, textbook)
3) It has a fixed compute-budget per output. (96 layers per token, from quantum physics to translation).
To make Language Models useful one must either:
Finetune after training (InstructGPT, text-davinci-002) to ensure an instructional context is always enforced...
...Or force the model to produce so called "chains-of-thought"[1] to make sure the model never leaves an instructional context...
...Or force the model to invoke a scratchpad/think for longer when it needs to[2]...
...Or use a bigger model.
---
It's insane that we're at the point where people are calling for an end to LLMs because they can't reliably do X (where X is a thing it was never trained to do and can only do semi/totally unreliably as a side effect of it's training).
Ignoring of course that we can in fact "teach"/prompt(/or absolute worst case finetune) these models to perform said task reliably with comparatively little effort. Which in the days before GPT-3 would be a glowing demonstration that a model was capable of doing/learning a task.
Nowadays, if a (PURE NEXT-TOKEN STATISTICAL PREDICTION) model fails to perfectly understand you and reliably answer correctly (literally AGI) it's a "serious indictment of its usefulness".
Of course we could argue all day about whether the fact that a language model has to be prompted/forced/finetuned to be reliable is a fatal flaw of the approach or an inescapable result of the fact that when training on such varied data, you at least need a little guidance to ensure you're actually making the desired kinds of predictions/outputs...
...or someone will find a way to integrate chain-of-thought prompting, scratchpads, verifiers, and inference into the training loop, setting the stage for obsoleting these criticisms[3]
---
"It will never be able be able to maintain coherence", cue GPT-3.
"It will never be able to do so without being spoonfed", cue Language Model Cascades.[3]
So what's next?
"It will never be able to do so without obscene amounts of data". Yeah for sure, and that will never change. We'll never train on multimodal data[4], or find a scaling law that lets us substitute compute for data[5], or discover a more efficient architecture, or...
LLMs are big pattern matchers (~100B point neuron "synapses" vs ~1000T human brain synapses) that copy from/interpolate their ginormous datasets (less data than the optic nerve processes in a day), whose successes imply less than their failures (The scaling laws say otherwise).
[1] https://arxiv.org/abs/2201.11903
[2] https://twitter.com/OriolVinyalsML/status/101752320805926092...
[3] https://twitter.com/dmdohan/status/1550625515828088838
[4] https://www.deepmind.com/publications/a-generalist-agent
GPT-3 is absolutely capable of understanding addition. It's handicapped by BPE's (so unless you space out adjacent digits the tokenizer collapses them into one token). But if you space out tokens and explain the concept, it gets it.
Bigger models get better at copying style with fewer examples. At some point, it'll be enough for a big model to see sufficiently few examples of real people drawing, "X in the style of Y" to get a sense for how to draw a thing in the style of Y.
Or an end-user could just feed a single instance of parody art in some artists style, ask for variations, in-paint what they want, and voila! Copyright free [citation needed] style transfer.
It will always be possible to launder style, even while adhering to promises to never touch an artists work (directly). Training on an artists entire body of work is not the only way (and thanks to copyright, a not even desirable) way to do so. edit: Open source models will also mean blacklists are unworkable.
Some of lucidrains other projects include the line "This is an implementation of thing with link" which I feel you'd prefer.
Like others, I too encourage you to open an issue to do the same here instead of leaving it till "after the fold".