martincsweiss

If you fine-tune a model with weights stored in bf16 and let torch.optim.AdamW write to them directly, you get interesting behaviour. bf16 can only represent changes to weights if the change is at least ~1/512 of the weight itself, so at small learning rates only the smallest weights can update.

As a result, you have a sparse fine-tuner that trains the small-magnitude weights and freezes the rest, with the cutoff set by your learning rate. This is a thing - I found half a dozen peer-reviewed papers (with thousands of citations) that fine-tune this way, but none of them reference this behaviour.

I dug into one called Quiet-STaR, because (a) it's a cool paper, (b) they released their weights, and (c) their code works! So I diffed their base model and saw that indeed the small-magnitude weights were the only ones updated during their training. Then I reran its recipe with an optimizer that applies every update, and watched it diverge. A higher-precision optimizer needs a 3 to 10 times smaller learning rate to get the same result, and I couldn't achieve a better result.

This started with Barron, Gnanapragasam & Tschannen's "A Fix for Softmax and Cross-Entropy" (Oct 2026). They show that the standard way of computing cross-entropy rounds the correct answer's gradient to exactly zero once the model is confident enough (past a margin of about 5.5 in bf16, which they call the precision cliff), and that this, interacting with Adam, is what produces the "slingshot" loss spikes seen during grokking. Fix the rounding and the spikes go away: something people had studied as training dynamics was rounding.

That made me wonder what else out there is weird like this. This is one, and it runs the other way: their rounding bug adds instability to training; this one hides it.

The code, logs and evaluation results are in mweiss17/quietstar-bf16-adamw, in case you want to check the receipts.

Author here, and Postdoc with Hugo Larochelle and Chris Pal.

I saw the LocalWiki last week, and it inspired me to make a learning tool based on "local" research papers. Many ML researchers write code that relies on a long history of research in ML and applied math. But which papers are you actually building on?

You can paste a GitHub repo and Standing on will read its Python code match pieces of it back to the research papers from which it arose.

nanoGPT, for example has 15 Python files and links to 13 papers including Adam, AdamW, LayerNorm, GELU, FlashAttention, etc. If you go back further, you get Adam and then AdaGrad (2011), Zinkevich's online convex programming (2003) and finally you wind up with Nash (1950) and Kakutani's fixed-point theorem (1941).

It's a very simple symbol table map with 4,800 classes and functions in torch, transformers, diffusers, timm, sklearn and others to the papers they implement. I used Claude and Codex to label the symbols and named each paper's parents, and check them against our ML research map, Lacuna.

We only support python and machine learning papers only for now. You can export to BibTeX, CSV and Markdown.

Yes I’d say so. That and a line of work on compositional generalization in deep learning.

I don’t think that’s where we are headed. Humans are the bottleneck but they aren’t replaceable by LLMs. We experience high fidelity multimodal observations all day long, generated by a 13 billion year old gigantic quantum computer. LLMs get a few quantized tokens.

The universe has too much detail for us not to contribute useful information to a human-so scientific process

Thurston wrote about this in “On Proof and Progress in Mathematics,” including how his own approach changed from proving theorems to helping a community understand the mathematics.

Nicolas Le Roux, a Lagrange Prize winner / prof / strong mathematician, turned me onto that around the time I finished my PhD in ML last year. I have been thinking about it a lot, lately, for obvious reasons.

At first, it felt silly to grapple with what science is only after finishing a PhD. This publication from OpenAI finally pushed me over the line to publishing an essay about this struggle between AI science and human science: https://news.ycombinator.com/item?id=50253759

Author here. I work on AI tools for ML researchers, and have had this essay rattling around in me for a while (since I completed my PhD last year, actually).

Seeing OpenAI's publication on "Working with academic communities to build scientific understanding" motivated me to finally post it.

I expect we’ll continue getting faster at producing results than at turning them into ideas other researchers can understand and use.

Academia already rewards new discoveries more than the work explaining and connecting them (I talk about this at length, and am working on a subsequent piece that goes deeper into the history of that).

OpenAI organizing groups to understand these results seems like a useful response, but I think that structurally the incentives and culture aren't there yet.

Yea pretty much. The picture I have in my head is of those arcade machines with the sort of snow shovel moving back and forth that sometimes pushes quarters off a shelf (called a coin pusher https://en.wikipedia.org/wiki/Coin_pusher).

You try to figure out where there’s lots of coins on the edge and drop your coin in to get it to push more coins out.

So some fields are ready to get pushed, and maybe density or patterns in concept space can show you that

yea probably - to be followed :)

edit: you can also certainly point a code agent at this paper and your library and have something interesting in a few hours. Or better yet, give your agent the Lacuna MCP and let it rip on the problem https://lacuna.tiptreesystems.com/mcp

We actually have a high-risk research idea for how to do this. It involves looking at the clustering as a vocabulary that compresses the concept space, and (one form) of significance as how much your paper / addition of concepts changes the clustering/vocabulary. I.e., if you change the optimal compression of the space, that's probably important.

The academic paper cost a few thousand bucks, max $10k in compute.

Lacuna is far more expensive and ~100x larger. If you knew what I know and started building it today (LLMs got cheaper).

Paper summaries cost ~$50k, concept extraction cost ~$25k, author profiles cost $25k, research proposals cost another $50k. So call it $150k for model calls.

It's also an online / updating daily so that costs too, plus infra, plus "spillage".

I had not, and I find it deeply hilaritragic that his paper is paywalled by JSTOR. Thinking of you, Aaron Swartz.

Hmm, interesting. Yea I think ML is a much smaller field, too. ArXiv + all the conference / applied ml stuff is ~150-250k papers per year. PubMed is 1.5-1.7 million new records per year, and like 40,000,000 total records, whereas in ML we only had to process a couple million. Plus, pretty much all of ML is open-access, whereas medical is ~50% closed.

If somebody gave us a $10-$20m grant for compute, and a few (dozen) helpful doctors showed up to help us QA / figure out how to approach it, it could probably be done.

Yes, vocab drift matters a lot. Most of the pipeline is basically compressing out any of the weird vocab/language stuff, so it doesn't have numbers and proper nouns and is in a sufficiently abstract concept space to cluster nicely. We use a density-based clustering method (HDBSCAN) which when you set min_samples=1 works as an online clustering (i.e., incrementally adding new things is the same as if you re-ran the clustering in batch mode).

It cost ballpark $250k

Thanks for digging into one! I had a project back on this back in my Probabilistic Graphical Model days with Simon Lacost-Julien (I believe this was my worst-ever grade in a course)... so I’ll defer to you on the mechanism.

I do know you're right that affinity/residence time and efficacy are separate axes. Bio is one of our sparsest regions (due to closed access papers, and our focus on core ML), so proposals there are more likely to have issues like this.

We've got some ideas on how to improve these issues, but I really appreciate you helping us by pointing this one out!

Necessity is the mother of invention.

I agree with you, though my own motivation is not centered on war, but (a long tail of) existential risks facing humanity, mostly centered around collective action problems.

Thanks for checking it out! We're working on a collaborative discussion / editing feature mediated by Althea (our AI agent). That should provide a way for domain experts like yourself to contribute to improving it. Hopefully we'll have that soon!

I'm working on a piece inspired in large part by William Thurston's "On proof and progress in mathematics"[1]. Should publish it next week.

Part of why math is effective is that its extraordinarily compressible. And you can take those ideas and apply them elsewhere (composition works well). The natural world has a lot of detail and is therefore hard to compress. That's why it's a good source of new ideas. Consider the relation between the McCulloch–Pitts neuron (MCP lol)[2], developed to model biological spiking neural networks, and an inspiration for modern artificial neural networks.

[1] https://arxiv.org/abs/math/9404236 [2] https://en.wikipedia.org/wiki/Artificial_neuron

Yann LeCun is right, sometimes

ya - I think my intuitions around velcro are firmly intertwined with my meat-space actuators and perceptual equipment.

In other words, I experience high-resolution multi-modal qualia all day and it's really noisy and I think a lot of new ideas and concepts come from obsessively exploring a small subset of those details. Maybe that's just me posthoc rationalizing my phd, though.

Very cool - the Alien Science work actually had a precursor in image generation. Cultural Alien Sampler: Open-ended art generation balancing originality and coherence https://arxiv.org/abs/2510.20849

I guess it makes sense that art folks like creativity!

Great question - the "universe" of venues we address is centered on peer-reviewed machine learning like ICML, ICLR, NeurIPS, COLM because (a) we have expertise in this area and so can therefore judge the quality and (b) it's all open access.

Therefore, the further you get away from the topic of ML, the less dense our coverage. Frankly, it's a tough to editorial choice to determine which research directions and therefore which proposals to include. ML is quite multi-disciplinary. And people who have a few papers in ML but many in (e.g.) medicine get upset when their profile is sparse. But we can't possibly produce good proposals in medicine due to its tendency to be closed-access.

I'd say science is somewhat fractal in this sense. A research area contains topics, those topics have sub-topics, those sub-topics have competing lines that "talk" to each other, etc. So if it works in core ML that means it's working for RL and deep learning and computer vision and NLP -- which is kind of cross-domain.

That said - we believe the method does need to have sufficient data to model the conceptual density. So it'll not work in closed-science areas.

One other thing I will share: we have been working on a way to enable verified academics with sufficient topical domain expertise to engage in small group discussions that update these documents. You can think of it like Althea (our agent) being an area chair and moderating a discussion to determine how to update pages. The reason we think this will work is that you can have an anonymous discussion with the AI system (and be blunt), but we can weigh your expertise when updating the documents.

Sure! My personal favorite I found at COLM a couple weeks ago: https://lacuna.tiptreesystems.com/hypothesis/passing-uncerta...

The idea is to allocate academic reviewer time based on the estimated quality of a submitted paper. But critically, you should maintain uncertainty bounds in your estimate. It's kind of like an UCB (upper confidence bound) bandit approach but inspired by cascaded speech translation (where you really should not throw away information at any point in your pipeline).

Thanks! I wish we could take full credit for the paper name, but it's based on a line of work from James Evans (Chicago-based computational social scientist) investigating "Alien artificial intelligences" https://x.com/profjamesevans/highlights

Oof - it's not like goodfire - we have quantitative results and a clearly defined method!

But yes, we might be bad at marketing. Internally the term we use more often is "concept elements".

The intuition is that if you take a complex treatment of a topic (like an academic paper), you should be able to break it down into some more abstract self-standing ideas. Then, critically, these units are composable and easily embeddable (i.e. they don't have proper names and numbers).

Here's a couple examples of these idea atoms pulled from our paper https://lacuna.tiptreesystems.com/work/the-alien-space-of-sc...: "Scientific ideation often conflates the logical coherence of a research direction with its likelihood of being discovered by the current community. This oversight makes it difficult to identify research paths that are scientifically sound yet unlikely to be discovered due to the existing distribution of expertise and collaboration networks."

Another example: "A generative model estimates the plausibility of concept combinations by treating conceptual units as tokens and calculating their length-normalized log-likelihood. By training on random orderings of these units, the model learns to identify which components are likely to co-occur in a valid research contribution regardless of their sequence."

Yea, so the way this was built causes regions of density (core machine learning topics) to be good and sparse areas like biology to be worse. There are some areas of intersection like Medical Coding, which anecdotally Joseph Paul Cohen (was a mentor of mine and a postdoc with Yoshua Bengio back in the day) reviewed while we were at COLM and he thought was good enough to share around his research lab. All that said, research proposals will be hit or miss.

The reason that biology is sparse is because we are processing full papers from open access venues. Machine learning papers have a long history of open access publication, whereas other areas still have a lot of concepts locked up. I suspect our method would probably not work very well if you only provided abstracts (like AI2 ScholarQA).

Our clusters and therefore our sampling of new ideas is based on density in concept space, and a lot of the key concepts don't end up in the abstract.

Go open science! All that said, the right partnership with a commercial publisher would probably make this work well in that regime.