Related:

100 Pages of raw notes released with the language model OPT-175 - https://news.ycombinator.com/item?id=31260665 - May 2022 (26 comments)

It really is great that Meta released the notes/logbook. Credit where credit is due. Very few other academic or industry labs release materials like this, especially when the reality is so messy.

Some interesting takeaways:

- Meta aren't using any software for scientific logbooks, just prepending a document

- So many hardware/cluster issues.

- Hot-swapping algorithms is common and likely underreported (in this case activation functions and optimization method)

- A well resourced team didn't solve enough issues to fully utilize compute resources until >50% of the total time into the project

If you are interested in hot-swapping, I highly recommend OpenAI Five paper: https://arxiv.org/abs/1912.06680. It includes 10 months log of how they adapted the model to releases of Dota 2 version 7.19, 7.20, and 7.21.

I agree hot-swapping is underreported, but there are some good existing reports. In fact, OpenAI Five paper today is probably more valuable for its details on hot-swapping than its details on main model which used LSTM and not transformer.

Weird they felt the need to update the version since they used a very limited subset of Dota anyway.

I wonder what software would be good for a logbook like this... I just use google docs for these kinds of things. Sure wandb and jupyter notebooks are good but they are not so good for notes and ideas and documentation

Sometimes the generic solution is just the best. No one requires special training on Google Docs and it just comes with handy features like version control and live updates.

Personally I use notion for this. Not dedicated software, but the extra options in formatting/structuring and linking can make a document like this a lot more readible.

We tried jupyter notebooks for this kind of thing once, and after the equivalent of ~50 pages it becomes unusably slow in my experience.

LogSeq? I use it for this kinda work...

This really does read like 'DevOps: Nightmare Edition'

> CSP fat fingered and deleted our entire cluster when trying to replenish our buffer nodes

Yikes

I have no knowledge of such things, but it seems they run Cuda jobs on about 150 nodes?

But why do they have so many problems to keep this cluster stable? Network failures? Bad GPU's? Bad drivers? Bad software?

Running fixmycloud and going after all those cryptic errors every day seems like a nightmare to me...

Seems kind of par for the course for an HPC cluster, no?

It makes sense to think of these things like Formula 1 cars, they are trying to eke out the absolute maximum performance, and reliability suffers because of that.

"Ordinary" cloud is more like a Toyota where you optimize for fuel economy and low maintenance.

This all makes me wonder: how reproducible is the final output model?

Not this one, but Google's PaLM (which is 4x OPT3) trains semi-deterministically.

These kinds of large transformers can be relatively reproduceable in results and benchmarks. However, making them converge to the exact same parameter set might not be a reasonable expectation.

> how reproducible is the final output model?

Not much, but it also depends on what you mean by "reproducible".

Do you expect a similar internal representation (weights) or a similar behavior?

the details of that specific model they ended up with? Irreproducible, unless the system was carefully designed and every detail required to do a fully reproducible computation was recorded and replayed. But they could easily produce a bunch of models that all sort of end up in roughly the same place and perform the same, ideally reducing the number of things they needed to change ad-hoc during the training.

Any particular highlights from this?

Honestly, no.