libraryofbabel

I know how LLMs work. I understand the output projection and logits and sampling just fine. I don't think we should call what is happening prediction because it is a misleading term with a lot of baggage associated with claims that LLMs are just statistically modeling language and "guessing" tokens without any internal representation of meaning. Since your reply is very short I can't tell whether you mean it in that sense or not.

And yes, I understand "prediction" is a standard term in the field. I just think it is a bad one and we should stop using it.

> it’s more like “predict the full output in full for this blob of input text"

Hm. I would actually go so far as to say "predict" is just the wrong word to use in 2026, as it drags in a picture of what LLMs are that's from the pre RLVR era, and which is frequently used to suggest they have all sorts of limitations that they've now surpassed ("an LLM will never solve a major open mathematics problem, they're just predicting tokens"). These days I tend to use the more neutral phrase "generate the next token"; I'd only say "predict" in the technical context of (say) the pre-training part of the training run where the object is indeed to minimize the cross-entropy loss function by predicting next tokens in existing text. Generating a token with a modern LLM is better described as something like, "given the totality of its training, including not just next token prediction with pre-training but also solving vast numbers of math and coding problems with RLVR, the LLM generates tokens one by one."

Anyhow. I'm not sure the weight understanding is all that strange. (It's impressive, but so are many thing LLMs do these days.) LLMs are very very good at coding and math problems, of which this is one, and in particular, their RLVR training presumably includes many tasks involving neural networks. It's amazing and impressive, I agree, but I'd say less amazing than proving major open theorems in mathematics, which we already know they can do.

Aaron is Hebrew and Erin is Irish. They just happen to sound the same.

You can go through it locally just fine; all you need to do is be able to run Qwen3 0.6B, which is a tiny LLM. A recent macbook (air or pro), or a PC with a good gaming GPU is perfect, but you can run the model and training examples on a PC CPU too; it will just be much slower. From his book (same examples as the video course):

> "the code examples have been carefully designed to run efficiently on a regular laptop, without the need for any special hardware. But if you do have access to a GPU, certain sections provide helpful tips on scaling up the datasets and models to take advantage of that extra power... The inference and evaluation chapters work well on CPUs and GPUs, while the training-oriented chapters benefit from access to a GPU if you want to reproduce the larger experiments."

So, consumer GPU helps but not essential. No need for expensive cloud GPU time.

I feel there's a misconception in a lot of the comment threads about what an LLM-from-scratch tutorial is for. Of course you're not going to come out of this kind of tutorial being able to train a useful LLM from scratch. (It requires too much compute, and datasets you don't have access to, and there are a lot of tricks and operational problems to solve for real training runs.) If that's your bar for "impracticality", then sure, this is impractical.

But that's not the point; the point is understanding these things better by seeing how they are made (or grown, if you prefer). Building something at a level further down the stack than you will ever go forces you to confront details that shape the layers above that you do deal with. This is one reason why Crafting Interpreters is such a beloved book here on HN.

LLMs are now at the core of tools we all use everyday... don't you want to know how they are made? Maybe not everyone has this itch to understand the platforms and tools they use, but I sure do, and it's helped me over and over again. Sure, LLMs aren't like operating systems or compilers. They're a quasi-organic mass of billions of parameters learned during training and honestly nobody (not even the mech interp folks) really deeply understands why they work so well. But that's no reason to completely give up and there is still a lot you can learn.

Take RLVR ("Reinforcement Learning from Verifiable Rewards"). This is the training technique that made LLMs get so good at programming and math in 2025 and 2026. A lot of people who confidently talk about LLM capabilities either haven't heard of it (and thus end up parroting out-of-date factoids about LLM training derived from the pre-2025 era) or have only a very dim idea of what it is. In this course you actually do RL training of an LLM to get it solving math problems better. You're right down in the details. It's interesting for its own sake but it will also give you a grounding for discussing questions such as: "why did LLM writing get so weird and bad in mid-2026?" and "why is it so hard to train LLMs to be good at things that aren't like math and programming?"

Anyway, Sebastian's two books are great and highly recommended; I worked through the first one a year ago and it was one of the best investments of my time in all of 2025. If you prefer video for learning this is great as well (although I'd still say - try the books first).

This paper has been bothering me all morning and I'm trying to articulate why. Some combination of:

* The consensus from the labs has been that Chain of Though ("CoT") monitorability is an essential tool for alignment. Sure, it's imperfect (models can exhibit deceptive CoT). But given how difficult mech interp is, it's the best we have.

* Remember all the noise made about OpenAI and "Looped Transformers"? All the safety and monitorability concerns there? And how OpenAI came out and made statements about how they are only doing a few cycles of looping (compute graph only 2x the size of GPT-4) and how they wanted to prevent "a race into unmonitorability." This is much worse.

* It's true that this architecture allows for "reading" the latent space reasoning traces by projecting into "nearby" tokens in latent space. But that means you're throwing out a lot of information that resides in the full vector, with no way to interpret it. You're just reading some sort of approximation. The LLM will still be reasoning with information you cannot see.

* Sure, there is a legitimate debate to be had about how useful CoT is for monitorability, whether the alignment fears are justified, etc. The most glaring problem this paper has is that the authors don't even really engage with that discussion at all. If there is an elephant in the room... maybe acknowledge, at least, that there's an elephant in the room?

* A little snarky, but this is a pretty obvious idea I think? Reasoning in latent space should occur to anyone who knows how a CoT model works, and there is plenty of prior work in this area. The reason people aren't doing this is because it cuts right against monitorability.

* This was announced on X in a series of messages that are clearly LLM writing. And which also did not acknowledge the elephant in the room. Sigh.

I am not a doomer, but I do not think discussions of alignment and safety are entirely vacuous. So when I see researchers who don't seem interested in that conversation at all, it worries me.

Instead of reading AI-slop explanations full of creaky metaphors, I'd advise you to just build your own basic coding harness from scratch. You will learn so much:

1) This old but still excellent blog post by Thorsten Ball is a good place to start (old in LLM years... written 18 months ago!): https://ampcode.com/notes/how-to-build-an-agent . It is written by a human, and gets across the core message really well. If you don't enjoy reading Go then just use an LLM to translate the samples into Python code or whatever.

2) Instead of relying on an api key like in the blog post in 1), you might want to use a local model that can do tool calls like Qwen 27B.

3) Use a frontier LLM for self-study and to help you write the harness boilerplace if you want, but ask it to force you to write the interesting bits by hand (e.g. the main loop structure, the tool call functions). Yes, coding by hand is still useful as an educational mechanism to force you to internalize some tricky concept. Be disciplined and don't slip into letting the LLM write the core logic for you.

4) Amaze your friends and colleagues with your newfound understanding of how agents actually work under the hood

5) Go even deeper and get into Inference engineering, LLM architecture internals, etc.

6) Profit. Perhaps? (Or at least enjoy the learning journey.)

Hmm. This piece makes a pretty strong claim with little evidence: that the recent drops in cache read pricing for new models from OpenAI (6.1 Sol) and Anthropic (Opus 5.5) are because those labs "shamelessly copied without acknowledgement" Deepseek's published KV cache optimizations.

Certainly anyone who knows something about inference is going to speculate, looking at the change in token pricing (and particularly how the % drop in cache read pricing is much larger than the % drops in pricing for other token types), that there is some kind of KV cache optimization behind these newer models. But even if is true, I don't think anyone can say with certainty what it may be. It may be the labs making their own innovations (they have some very smart people, and this is probably an area where having unlimited pre-release access to frontier LLMs like Fable and Astra gives an additional research edge), it may indeed be the direct application of Chinese labs' methods, or it may be some combination of the two. Sure, it is fun to speculate about, but beyond the facts of the token pricing changes and the increased inference speed, it's just speculation. The certainty the author displays here is not very helpful.

The author also seems to have a bit of an axe to grind agains the US labs, judging by the tone. I think that detracts from the discussion too.

They also seem confused about why Chinese labs have released these optimizations recently. Well, you have to release them (with or without explanation) if you are going to release an open weight architecture, and that is what the Chinese labs have been doing for a long time. Sure, there are reasons behind that to discuss too, but this isn't exactly new.

So, this is an interesting topic to think about, and the Chinese labs do indeed deserve credit for some very clever new attention and inference techniques, but I would read it with a skeptical eye.

This is one of those papers that ought to have several textbook chapters unpacking some of the insights... I'll just pick a couple things I love:

1) There's a kind of "rabbit–duck illusion" moment where they show how you can reframe all the linear algebra around attention in a totally different way, demoting the Q, K, and V matrices in favor of emphasizing a set of much larger matrices that are mathematically equivalent and very useful for interpretability purposes. (So, don't trust anyone who insists that vague analogies about "keys" and "queries" are the only good way to understand attention! They probably haven't read this paper.)

2) They show the importance of the "residual stream" inside LLMs: it's not just a series of bypasses of layers that's useful for training stability (like I first had it explained to me) but a kind of main communication bus that runs unbroken through the whole model from embedding to output. Attention heads and Feedforward are off to the side, adding embeddings into the stream. Somehow I find this a much more satisfying way to understand LLM architectures, and whenever I see an architecture diagram now I mentally redraw it with the residual stream at the center.

This book (and the book author's blog) is the best source for learning LLM internals for someone just getting started. It's clear and well-written (by a human!) and goes into much more explanatory detail than this does: https://sebastianraschka.com/llms-from-scratch/

Thanks for clarifying. You're right, I skipped over talking about dynamic looping, since it adds another level of complexity to the discussion, and OpenAI's claim (quoted in TFA) that the compute graph depth of Astra is "within a factor of two of GPT-4" basically denies that they're doing it for more that 1 (or maybe max 2) dynamic loops. And that is equivalent to "stack repeated layers a couple times, but with dynamic off-ramps."

The ability to compute any computable function between tokens given an ability to loop an arbitrary number of times is a nice theoretical point, sure, but ultimately if people are still using single digit hard cutoffs on the number of loops, I'm not sure it's all that important.

So, I agree it's right to say that arbitrary length dynamic looping could open the door to making monitoring very hard indeed, by extending hidden states further and further. But I would speculate that if it actually worked better than extending the sequence with CoT tokens, we'd already be seeing it in strong open weight models. It's a fairly obvious thing to try. And we're not seeing it, AFAIK. So I do wonder whether it's something we really need to worry about in practice, compared to all the other things we have to worry about.

> that effectively moves the CoT inside the architecture

This may be a bit of a nitpick, but... does it? I agree that giving the model decisions on looping certainly makes interpretability harder, because it adds more transient internal states to deal with and changes the number of them depending on prior states. But is it really pulling CoT inside the forward pass, if the sequence length it's operating on isn't growing? In some sense the whole technique and tradeoff of CoT is "add more tokens to the sequence, use them to reason with", with one of the benefits being, you force the model to output tokens, so you can (hopefully) understand it. And the big point TFA is making is, nobody is doing recurrence over sequence length as far as we know.

Well sure, that's the possible weak point in Sebastian's article: it could be true that there's some more sophisticated stuff going on in Astra around looping, because OpenAI haven't specified their architecture. But it's always been true that, since we don't know what's in their black box, there could be arbitrary amounts of innovations inside the models that we could speculate about. So the question is, does knowing they use "looped transformers" really add any dramatically new information that we should worry about? And what this article is saying is, not really, because the mostly likely pattern that's referring to is just, effectively, stacking layers and reusing weights.

> And by Open AI's own admission, Astra's CoT is significantly harder to monitor and it exhibits a significantly greater control over its own CoT than any other model released.

Oh sure; I don't think anyone is denying that larger issue? But does it have anything to do with looping?

Everyone interested in LLM internals should read Sebastian. He's great.

The tldr here is that the recent "The Information" article[0] reporting GPT 6 Astra was using “recurrent depth” or “looped transformers" made it sound like it was some special new scary thing ("secret technique!") that made train-of-thought monitoring harder to do. In fact, it's just the same as stacking more transformer layers, except that you reuse the weights and so save GPU memory. It's still just producing one token at a time, and the token sequence positions aren't interacting in any "recurrent" way that's different from a regular LLM architecture.

So, you can still monitor train of thought with these models just fine... well, if you're OpenAI, anyway. Users haven't been able to see an unsummarized trace since o1 days, because the labs are worried about distillation of their models by Chinese labs.

(There are some legitimate interpretability concerns about stacking transformer layers endlessly, but we're known about that for a long time. And the "looping" here isn't really the source of any new issues here, except insofar as it's a cheap way to add more layers.)

[0] https://www.theinformation.com/articles/secret-technique-beh...

> This makes the total effort linear over the entire context (or constant per forward pass).

This is incorrect. The compute required per forward pass to generate each additional token during decode will scales as O(N), even with a KV cache (without a KV cache, it would scale as O(N^2)). Over generating N tokens, it's O(N^2) with the cache (and O(N^3) without).

It's O(N) for a forward pass because that new token still has to "attend to" to each previous token. That requires N dot products: between the cached key vectors and the new query vector for the new position. You also have N reads from memory (K and V) which is probably gonna be your actual bottleneck. (Decode is memory-bound.)

This is why you should avoid long contexts, if you can, even with a warm cache. You will get charged more, in "cache read" tokens.

I once heard someone describe Admiral Cloudberg's air crash investigation blog as "the best thing on the Internet" and I still think that might be true. She is a great researcher and a superb writer, which stands out all the more these days in these fallen times of load-bearing AI slop. Her blog is beloved of both aviation geeks and, for obvious reasons, SREs (and other engineers).

She recently posted what's basically her magnum opus, on the Potomac river midair collision.[0] It's the length of a short book. I'm working my way through it; I'm taking a flight today, and I'll probably read it on the plane tonight.

This article, on how an A380 with 469 souls aboard almost crashed, but didn't, is one of her best. I like it because it's a happy ending, and a story of the combined effects of good engineering design (the astonishing degree of redundancy built into the A380 proved just enough to prevent catastrophe), and human troubleshooting under pressure in the cockpit by some smart people with vast experience who remained calm throughout. (Another favorite article of her, on the Air France 447 crash, tells the exact opposite story: poor systems design combined with pilots basically losing the most basic ability to aviate in the heat of the moment. That one is a much darker read.[1])

[0] https://admiralcloudberg.medium.com/reaping-the-whirlwind-in... [1] https://admiralcloudberg.medium.com/the-long-way-down-the-cr...

> there’s no way to “crack open” an LLM and see precisely where each skill or tendency lives

Mechanistic Interpretability has entered the chat.

For a classic example, see https://www.anthropic.com/research/tracing-thoughts-language...

The spirit of your point stands, though. This kind of research is interesting to read about, but it's very hard, more like neuroscience or biology than computer science ("LLMs are grown, not made"). You're dealing with a lot of extremely _messy_ complexity, for which organic life is really the only good point of comparison. Most of us here are't really equipped for that kind of work; it's not at all like, say, reverse-engineering a piece of software written by humans. And of course the only people who can do it on frontier models from Anthropic and OpenAI are people within the labs themselves. (But I'm optimistic we'll see more of this work on open weights models...)

I am saying this probably is "silly behavior by a government" and it is a milestone that points towards what the future may look like. Why can't it be both?

It's easy to wave this aside as the current administration playing political games. But I don't think there is any reason to assume that the current era of open availability of models is going to continue indefinitely. Do you think that Chinese labs will continue to release open models forever, even why they get to the level that Mythos is at now, and beyond? And do you think that a competent US government would have no interest in regulating and restricting model access in 2 years time, assuming that model capabilities continue to improve? I think we bias towards thinking the status quo is the norm and will continue, but this news invites us to question that assumption and think about different ways the future could go.

You may be right, and I actually agree with you: I think that in this case the most likely outcome is that Fable becomes available again at some point, albeit possibly only to a restricted set of users within the US.

But I think my larger points stands: even if we do see Fable access again, this is the beginning of government restriction of LLMs and we are going to see more and more of it. In fact, I would be very surprised if we ever see an open weight model with Mythos capabilities. Chinese labs have been consistently releasing open models 6-12 months behind the frontier. In 6 months we may see them go dark.

Similarly, in the US I think we can expect more and more government restrictions on the strongest LLMs, in ways that may go beyond flimsy checks like uploading a valid US passport. It may not happen this year but I think it will happen eventually.

It still surprises me sometimes that LLMs are just available for _anyone_ to use. Isn't it odd that it turned out this way? When I grew up reading sci-fi I thought AI, if I ever saw it in my lifetime, would be something locked up behind the walls of big corporations and governments. But instead we have all been able to use it for an infinity of banal purposes for $100 a month. This is a strange situation but we have got used to it. But it may not continue that way.

So many comments here missing the big picture, and just gleefully pointing out that Anthropic got what they deserved, or that this is the natural culmination of some kind of marketing stunt.

The real story here is that this may be the beginning of governments restricting the availability of strong LLMs to the public, to you. Fable was the strongest model on the market, and the US government has told you you can't use it (technically, only if you're not a US citizen, but in practice, even if you are). If you think the solution here is going to be open source Chinese models and / or running on your own hardware, think again. Do you think China is going to allow the strongest LLMs from companies within its borders to be open source a year from now when they have Mythos capabilities, if the US government is keeping the strongest American models back? Unlikely. These are heading in the direction of being powerful cybersecurity weapons and it will be in the interest of nation states to restrict and control them. In 2 years time, I would be surprised if the strongest LLMs are available for general use at all.

Will we be the poorer for that, or will we be safer? I think poorer, because I hate being told what technology I can and can't use, but I'm not certain. Maybe you think the government should restrict strong LLMs. Maybe you don't. But either way, this is big news and a rubicon has been crossed and a precedent set. That's true even if the motivation for this is just the government settling scores with Anthropic.

Where? Please point it out! All he says is there will be more desperate people on the market in future and because they're desperate they'll have to accept trial work. But that's not answering the objection that people who do still have stable jobs in that world won't want to interview at your company if you require them to do a stint of trial work first. And those people include a lot of desirable candidates.

Well yes, there is tons of AI bullshit about and all sorts of scammy behavior, but I don’t think that says anything at all either way about whether the core technology is a “scam”, theranos-style. In fact I’m not sure how it could be otherwise: of course there’s going to be all sorts of hype and scamming around a novel, rapidly-progressing and potentially transformative tech like this, even if it works.

If you want an analogy, look at the history of the early railroads. Full of hype, bullshitters, scammy investments, robber-barons, unrealistic promises, and with their own legion of naysayers at the time. Yet the core technology worked and it did transform the world in the end.

People that know more about nuclear physics than I do already answered, but I’ll just say that:

1) It’s easy to think about the past in terms of what we now know, and it involves a real effort to put yourself in the shoes of the people living at the time and to imagine the “fog of war” in what they knew. In 1945 nobody had ever tested a nuclear explosion before and there was still all sorts of uncertainty about it. And as one of the other commenters pointed out, in particular there was a lot of uncertainty about how fusion worked.

2) The center of the Trinity fireball did in fact produce hotter temperatures than had ever existed on Earth before. Temperature and energy being different things.

In some sense the final experimental proof that a nuclear explosion would not set off some unanticipated new chain reaction that would destroy the earth - unlikely, but hard to completely disprove - was Trinity itself. Only after Trinity is it obvious and completely proven how the physics actually worked and obvious that there were no additional reaction pathways that got missed. That is a disturbing thought.

You’re thinking of the other bomb, the U-235 one, which they didn’t test at Trinity and which was dropped on Hiroshima. That is two separate pieces of Uranium that are slammed together to create a critical mass. The Pu-239 core was a single sphere of metal. It was subcritical until you compress it down with a spherical implosion from explosive charges all around it (from the size of a grapefruit to the size of a lime), at which point it reaches a high enough density to go critical.

That’s the one I meant. It’s the core, but in a box, which makes it look even more innocuous, like he is indeed just lugging a piece of industrial equipment around. There’s lots of photos of the actual (unboxed) cores online if you search.

I used to teach a class on the history of contemporary science (WW2-present) and I started the class with Trinity. There’s no other moment better.

We know how it turned out, but the people there waiting for the test did not know how it would turn out. The bomb might not have worked. Or it might have ignited a fusion reaction in the atmosphere and destroyed the world. Hans Bethe had sat down and done the calculations on that exact scenario and said it would not, but there was always the possibility of missing something. Enrico Fermi was offering bets on it on the day of the test, as a dark joke.

In the end it worked as expected; one of the most successful and horrifying experiments in the history of science.

Of all the photos from the test the one that struck me the most looking through them today was the photograph of the plutonium core being carried into the ranch house for assembly in a little heavy box. It’s a small thing, about the size of a grapefruit, although twice as dense as lead. It looked just like a sphere of any old metal, but it was something profoundly alien, made inside nuclear reactors. And it still is so strange to me that something that small has so much energy locked up inside and that, by imploding the little sphere just right, we can let the demon out.

Trinity is one of the pivotal moments in the history of our species and eighty years on we still don’t know what the eventual consequences of it will be. The bombs are still here waiting for us and they still pose all sorts of terrifying questions for the future that most people prefer not to think about.

> We haven't seen a significant increase in the quality of LMM output since 2023 that hasn't been the result of throwing even more energy and compute at it.

This is completely false. Most of the dramatic improvements in LLM quality in the last two years were due to the application of new post-training methods, especially RLVR. It’s really interesting to read about (you should!) and it is the whole secret to why LLMs did not plateau in 2024 or 2025 like many people confidently predicted. Sure, RLVR requires compute to do, but this is not just throwing more compute at 2023 LLMs.

> Looking away shall be my only negation.

I’ve been thinking of building myself my own frontend to HN that makes it impossible to view comments, for this reason. Yet sometimes there are still really interesting discussions and it’s hard to let go of what for me feels like the last social media I want to be part of.

Well yes, but there is a choice being made here and I would love to believe we can do better. The rational response to being afraid about your livelihood isn’t to spend time filling every HN thread on LLMs with embittered negativity. Not to mention all the flat denials that LLMs can do mathematics and write decent code, which is almost a self-contradictory position if you are worried they are going to replace you.

There are a lot of big issues at stake here and just because a person is interested in what AI can do and curious to discuss it does not make them uncritically positive about it’s effects on society, the economy, and the world. Yet that is often the assumption and it leads to battle lines being drawn, on every AI discussion, over and over again. It means the serious discussion gets swamped and that makes me sad.