D-Machine
I dunno, really need to see the comparative performance of other classifiers and the base rates in these datasets to know whether these performance metrics are anything we should care about, and whether e.g. a classifier on simple verbal probes to the subjects might simply yield better predictions than going through the EEG signals. You can do some really interesting EEG biofeedback stuff, so maybe this is useful, but I would think the full EEG head-gear would also provide much richer and more useful information than a measurement based on a single chip. Presumably this would be for psychiatric disorders or brain injuries where the core dysfunction we know is limited to a very specific region(s).
This is entirely speculative and highly disputed, even if you are doing the more sophisticated thing here and by Language you mean something like mentalese.
Much cognition is embodied and makes use of visual, tactile, auditory, kinaesthetic, and other sense modality representations. "Thought is mostly a Language" has been a wordcel kind of argument with little empirical or philosophical support for a couple of decades now, IMO. Language is how we communicate thought, not primarily how we think.
Not that this refutes your main point: LLMs are trained almost entirely on (written) language, and since language is only a single, limited aspect of real cognition, this makes them so annoying and alien to read.
Most importantly, you rarely have "cognition-free" emotional states, because emotions always involve some mental model or representation of the self / world / self and world, i.e. they involve a cognitive state. Only in very extreme cases (drugs, perhaps some cases of persistent depression, etc) might you be able to have emotions that are largely uncorrelated with any kind of particular cognition.
Likewise, the same is almost certainly true for cognitive states, in that we tend to think about / model things because we need to evaluate those things. Evaluation too seems to rely heavily on emotional processes, in that damage to emotional processing tends to make rational / logical decision-making severely impaired (e.g. Damasio's work in the 90s, or https://pmc.ncbi.nlm.nih.gov/articles/PMC2396787/). Regardless, you can't remove evaluation / valuation from most thinking, and that is fundamentally linked to emotion.
Doesn't mean you couldn't potentially build a machine / thing that detects mental states, mind you. But it likely won't be very useful if it is just using simple folk-psychological terminology and concepts (or using such concepts will likely not result in much insights into the machine outputs).
Yeah this I know, by "bouba" being opposite I meant the waveform.
Really the phonetic terms here are not actually relevant and have betrayed me. It's more that the 'k' sound is "more [ex]plosive" in that it seems to build up more pressure and turbulence (probably as you say, because the consonant is in the rear, so pressure builds and bounces around in the mouth more, whereas the "b" just mostly blasts out the front more quickly). And the "k" surely has a more peaky / spikey waveform in the audible range.
Lol, true.
If you had actually read the article, they explicitly tested for and controlled this...
It is a bit mysterious why animals would all map e.g. sharp auditory waveforms to sharper / spiky lines (kiki figure), and then smoother sounds to smoother lineforms (bouba figure), but, reasonably, these senses evolved together and used far more shared underlying processing machinery than they do presently. After all, both light and sound reach us in waves and both carry energy.
Some primitive mush-blob perhaps just started processing all incoming energy waves together, differentiating e.g. smoother vs. more staccatto waves before making the distinction between e.g. sound vs. light. Or I dunno, but IMO it makes more sense when you consider evolution, and don't just look at processing now in isolation from the likely development of these senses.
E for expectation, blackboard bold, double-bar norms and indexed max/min are all early to mid 20th century notation, making them thoroughly modern. Only summation notation dates since the eighteenth century, and its relative stability over that period is in fact rather a strength and not a weakness. Summation notation is also nearly universal in any quantitative field, and far, far more concise than the equivalent looping C-style pseudocode or functional-style pseudocode.
> You're also connecting the syntax to understanding of the concepts which is the cherry on top of the math snobbery in this thread
Because that is in fact a big part of how it works. Good notation takes a lot of time and thought to develop, and the notation used in the post is far, far more concise and packs far more information into a tiny amount of space than pseudocode or other programming conventions. Much of math is abstraction (compacting complex ideas into specific symbols or concepts, and then coming up with simple rules to allow their manipulation), and is possible in large part only because of good notation like this. See some of Terry Tao's thoughts on the subject below.
The notation here is really also all first-year introductory notation, and incredibly simple. Most CS or computer engineering students should have seen it, and anyone that thinks they can learn anything non-trivial at all about neural networks by avoiding such basic notation is simply not a serious person at all.
EDIT: And it is a mistake to imply pseudocode is generally superior or easier to understand, because pseudocode is often too precise, and implies more structure and relations and implementation details than are actually needed or important. Meaning it actually often communicates the wrong thing compared to good notation.
https://terrytao.wordpress.com/advice-on-writing-papers/use-...
https://mathoverflow.net/questions/366070/what-are-the-benef...
Also make sure to understand how self-attention layers are actually applied and the exact math and matrices involved. E.g. in PyTorch you are going to be doing sa(x, x, x), where x is the previous layer's output (essentially, embedded tokenization), and sa() does quite a few matrix multiplications. In cross-attention, you are doing sa(x, x, y), where y comes from a different part of the architecture.
Basically, read broadly here and don't get stuck on a single analogy / intuition for what these layers are doing. Focus on learning the actual math of the layer and the network first, and then look for analogies that help pump your intuition. But don't try too hard to force a single analogy onto the math.
EDIT: Also modern AI can provide useful insight into posts like this, I highly suggest using them when learning these things. E.g. https://chatgpt.com/share/6ad3aa3d-d110-83ea-a8c5-8653b34743...
Embarrassing to complain about notation so simple, and, frankly, if you can't understand math concepts and notation this basic, you simply can't meaningfully understand modern neural networks: without the math, all you are doing is substituting loose analogies and metaphor for understanding. This is clearly an article for people that want more than that.
Mainly, since self-attention layers operate on deeply-stacked representations of the outputs of previous layers, and since training has a loss function that doesn't make the updates work in the way the article says, the analogy to memory and lookup tables breaks down quite strongly. IMO, thinking of Q, K and V as memory or key-value lookup is thus not actually very helpful for understanding what these layers are really doing or why they work. Others agree (e.g. https://news.ycombinator.com/item?id=46535412), so don't feel bad if you think there is something hand-wavey about the lookup-table and memory explanations - it is only a weak analogy.
This has come up on HN before, and probably it is most accurate (but less intuitive, sadly) to see self-attention as a generalization of kernel methods (https://news.ycombinator.com/item?id=46535549). Broadly, self-attention is just one of the simplest ways to model similarities / correlations between all inputs (i.e. kernel method), but there are many, many other attention mechanisms. There are also many other ways to implement memory (e.g. LSTMs), and many other architectures that might in fact have even more memory capacity than transformers.
That being said, this is still truly an excellent article and it really provides some great intuitions for why transformers might work as well as they do. The softmax / exponentiation here functioning to increase memorization capacity by suppressing weak activations is super clear and interesting, and I really appreciated it. I suspect is not really be what is happening or is actually important about it, given other research (https://arxiv.org/html/2410.18613v1), and because much of what works in DL is usually more about matrix conditioning and keeping gradients well-behaved generally, rather than what the actual bare final math / weights in isolation from training would tend to suggest. But it does give a great intuition for why a trained transformer can have the capacity it does.
Just if you curate your Twitter feed, you gain very little extra from BlueSky or Mastodon, and if you choose only the latter two, you get a much more biased and limited set of views. (Twitter has plenty of crazies from all parts of the political spectrum, but BlueSky and Mastodon run mostly left, and have much, much smaller user bases). HN also is extremely limited in what it discusses, so it doesn't make up for what is missed by ignoring X either, IMO. X is still one of the most efficient general public squares by a long shot.
If you can't stand Elon and don't care about what you're missing on X though, I can fully understand just not bothering with it. You aren't missing much.
Yes, partly. But it also refers to complexity arising from the component parts that requires additional tools (formulas, descriptions, etc) to meaningfully describe. Or at least, emergentism has always formally involved more than mere predictability from parts (https://plato.stanford.edu/entries/properties-emergent/). I am talking about weak emergentism here, so no need to get into metaphysics and mysticism.
> My claim is nothing in LLMs is emergent in the sense of being inexplicable from an ideal consideration of simple/linear aggregations of its neurons.
IMO this contradicts even your minimalist definition. Almost no one predicted simple LLMs would be where they were today, and you can't currently predict their behavior by looking at their weights.
Something doesn't cease to become emergent if you can roughly explain it by making references to its parts. We can broadly explain how human behaviours result from drives + sensory processing + associations and etc., but human beings are not very predictable in many cases, and also exhibit consistent behaviors that tend to be very hard to predict (or even explain) from basic knowledge of the brain, drives, and body. LLMs are no different, and what they are doing today couldn't be more clearly emergent, even by your definition.
Also, something doesn't cease to be emergent even if you can perfectly explain it with reference to the parts. E.g. attractors are emergent behaviour, even though they fall out of very simple formulas and so in a sense are completely due to their defining parts. But describing the shapes of e.g. the Mandelbrot set or the Lorentz Attractor requires more than the base formula that generated them. So they are classic examples of emergent phenomena. Likewise, a person's intelligence might one day be fully predictable from biological and genetic knowledge and measurements, but it wouldn't mean that intelligence stops being an emergent property that is still observable and useful to use itself in other models.
You can for example argue that an LLM has no emergent drives (https://xxcancel.com/jon_stokes/status/2111158233993797870#m), and that would be a coherent claim, under a certain definition of "drive".
But your argument here I cannot follow clearly, because LLM behavior is clearly emergent in any sense of the term, including yours. EDIT: Or basically, LLMs show weak emergence, but if you disagree with this, then it would seem to me you perhaps disagree with the concept of emergence at all.
You'll need to elaborate here, because Navier-Stokes plus Mathocalypse reactions are extremely strong evidence for coherent emergent behavior coming out of these models.
> The only thing emergent in LLMs is degenerate state leading to a breakdown in tracking relevant features of the input and a reduction in quality of output.
This is so provably wrong with two seconds of Googling / AI-asking that you must be using some completely idiosyncratic definition of "emergent" here.
> I expect this has as much to do with our lack of precision in our tools rather than with the concept in principle.
Unfortunately in this case it is the exact opposite.
Nothing you said responds to anything I said.
Rice's theorem (which you are clearly citing, but won't clearly mention, for some reason) has almost zero actual practical relevance to AI explainability.
You make low-quality and low-effort AI-denialism posts whenever you pattern match "all non-trivial semantic properties of programs are undecidable" to... anything you think matches the folk/common understanding of those terms, without considering how irrelevant these statements are to actual real-world applications and engineering.
You ask for links, but post nothing of substance and post no references or even mention relevant details for your own claims.
https://xxcancel.com/NeelNanda5/status/2110875974296654062#m (and the re-post on LessWrong: https://www.lesswrong.com/posts/x4kvabkF2Jk8GLZp8/linkpost-n...)
TL;DR:
This is interesting work, but I don't think it's a significant update to the big picture of interpretability. [...] I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.I agree.
> Modern researchers obviously want the goal of simplicity & composition, but to arrive at that, they build abstractions like transcoders and SAE's to get into their pet ontology (with the assumption that neurons themselves are garbage to understand). This obviously raises the question - why privilege your basis?
Agreed, and IMO the implicit reason for privileging here is supposed to be "consistency with existing human understandings and concepts", and this is why we get short pithy things like "V-shaped" and lots of pictures showing relatedness that we all can see and agree on. But what if the domain of inputs is NOT something we understand (e.g. the whole of mathematics, or something new or that we don't have any understanding of)?
I called this a validation problem in another comment here (https://news.ycombinator.com/item?id=50108184). I.e. if we apply whatever methods Nick is using to "explain" a model on a domain we don't understand (like a deep model trained only on fMRI and EEG data of the whole human brain in various contexts), we aren't going to be able to tell if that description is correct (or just better than any other description), because there is no simply visual / common-sense inspection of the description.
> The resulting interactions between said neurons may be incomprehensible, but the point is that this provides a layer of objectivity
Agreed, emergent behaviour is clearly the case and limits how useful understanding individual neurons can be. IMO this is especially the case because layers are (weighted) summing generally millions of contributions from previous neurons.
> But if he's right, then you do actually have this privileged base layer to work with, even if you don't know what the base layer means. That's pretty important, actually. Assuming this holds up in reality, ofc.
Yes, it not being "chaos all the way down" is good, actually. And this post shows there is indeed a good amount of order hidden away if you look at in the right manner.
But I agree the author is obfuscating weak claims via length / complexity (and it may be as much for his own sake as for his readers'). The basic claim ('maybe AI will be more explainable/tractable than initially thought') is IMO intelligible, and not completely nonsensical, but also extremely vague and vibesy, which, technically, I must agree is in the same territory as nonsense, at least.
Based on our other exchanges here we mostly agree with the problems of the article I think.
Not really, because of emergent behaviours and also because there are still millions of neurons being summed (with variable weighting depending on inputs) at each layer.
> and even constrain its space of behaviors by activating/suppressing a small set of neurons that have very straightforward effects on its behavior.
You can influence behaviour by doing this kind of thing, yes, but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).