RaffaelCH

If you haven't seen it, this recent blogpost might interest you: https://news.ycombinator.com/item?id=50105647

It argues that our understanding of neurons storing information in superposition was just a limitation of our tools and us thinking about it wrong. Instead most neurons initially specialize, and then branch out into related concepts. In addition, understanding one part of the network helps better understand other parts. And with agents we can scale this to understand non-trivial networks.

While, if true, this would more directly help with interpretability, a better understanding could also help with improving NNs in general.

From what I understand, to work with copyrighted books they need to essentially format shift (i.e., scan and destroy the physical book). So a book vault would not solve this issue.

A book vault would still be useful for out-of-copyright works, but this would only cover a (probably relatively small) portion. Also, I'm not sure how easy it is to reliably determine copyright at scale, so they might just decide that it's not worth it.

At this point my only hope is that in the long run these scans make it to the public somehow (leaks, copyright changes/expiration, whatever), where they can then be accessed and preserved by everybody. Then we could have our true digital library of Alexandria.

Regarding your first point, if the cost of not doing it while you wait exceeds the cost of training/maintaining the model, then this approach still makes sense. This really depends on how fast you expect cheap (and open-source) models to improve. So it might only be a temporary strategy, but still worthwhile. Also, I expect that specialized models will always be better (cheaper or better outcomes) than a general model, similar to how specialized HW like GPUs is still used even though CPUs have improved a lot too.

Maintaining in the way of ongoing training is relatively cheap, considering that the initial training run is only 500$ (in this example). Creating new examples to account for drift in the training data is more expensive, but these examples can then be reused when training a new model. And to some degree you need them anyways, to evaluate the models and prompt changes you make.

Overall, as you pointed out, this won't always make sense (either due to the cost of creating examples and training, or simply the lack of available data). But they point this out themselves in the diagram towards the bottom of the page: it only makes sense for frequent and verifiable tasks. This might result in this only being a sensible approach for very large companies (they talk about millions of decisions), but it might make sense for them.

> Then they feed it the original software (AFIK).

My understanding is they did do the dance. From the article: "He fed only the API and the test suite to Claude and asked it to reimplement the library from scratch."

One could still make the argument that using the test suite was a critical contributing factor, but it is not a part of the resulting library. So in my uninformed opinion, it seems to me like the clean room argument does apply.