That's an interesting question. It actually provides a nice way to parallelized training: Pretrain e.g. the first 3 branch levels, which effectively fragments the model into 8 separate parts, which you can continue training across 8 independent servers/nodes with no further communication between the nodes. A central server would run the 1st 3 levels and mark parts of the training set that each node has to train on. Maybe you could do this for the whole network and distribute the training in SETI@HOME style all over the world.

Hold on, you don't even need to freeze the branches completely: each node could train 1 branch on the path to its leaf node and communicate a change in the branch node to a central server, so you can distribute training without having to pre-freeze the branches. Still would need some pre-training though, and the splits would change slowly, and the attention mechanism could complicate things.

Currently distributed neural network training SETI@HOME style looks like a complete pipe dream that nobody is taking seriously. But a smart branching mechanism like this could suddenly make it possible. Folding@home reached 1.5 exaflops, which made it the world's largest supercomputer. Imagine the models we could train that way, they would far surpass whatever OpenAI or Google could train and would be public.

This!

If this becomes true then it’s a game changer. I hope you are correct.

Also steps up the economic benefit of, and therefore demand for, botnets. We really need a solution to bad actors controlling vast amounts of compute.

That ship has sailed, and her name is bitcoin.

If bitcoin keeps the botnets away from world beating AI, worth it.

I am barely understanding, so a stupid question:

Does this also mean that it would be possible to train on parallel GPU-poor setup instead of needing lots of GPU memory / bandwidth on one computer?

Probably not. The paper talks about this as an inference time optimization

Apologies for layman question: how much tera/peta/exa-flops do current models use to train?

Well, I'm assuming they'd use whatever they're given, so maybe the question should be "how much less time would training take on a 1.5 exaflops computer?"

As many as they can afford.

A lot of clusters are totally homogeneous, at least within some very large domains, so for a given interconnect and a generation of GPU you know the maximum message latency, the peak sustained pflop rate, and so on but what often matters is some combination of the depreciation-cost-per-time and the watt hours per unit time, where you can sort of approximate both if you ignore the unfortunate realities, which then act as a multiplier.

For example, a problem is network issues - and not just scale - as the training sequence often involve billions of cycles of short compute-sync sequences which are bursty (e.g., all-to-all, barrier, compute, barrier, all to all, ...) but between which there isn't enough time to engage low power modes so you're burning $ due to slack and waste. This is true in different ways for a lot of training approaches.

You can approximate this, but it's so sensitive to data set size, specific training schedule, etc. that you won't be able to get the most important answer.

You should check out Hivemind[1]. It is very similar to what you described except it used MoE for "fragmentation". They have a couple of examples of pre-training in their repo. Hivemind was also used to build Petals[2] but it only supports fine-tuning and inference[3] afaik.

[1] https://github.com/learning-at-home/hivemind [2] https://github.com/bigscience-workshop/petals [3] https://chat.petals.dev/

Why not have a GitHub or hug face demo instead of saying it’s the best thing since slice bread?