You gradient descend on your state.

Each step needs to work on up to date state otherwise you're computing gradient descend from state that doesn't exist anymore and your computed gradient descent delta is nonsensical if applied to the most recent state (it was calculated on old one, direction that your computation calculated is now wrong).

You also can't calculate it without having access to the whole state. You have to do full forward and backward pass and mutate weights.

There aren't any ways of slicing and distributing that make sense in terms of efficiency.

The reason is that too much data at too high frequency needs to be mutated and then made readable.

That's also the reason why nvidia is focusing so much on hyper efficient interconnects - because that's the bottleneck.

Computation itself is way ahead of in/out data transfer. Data transfer is the main problem and going in the direction of architecture that dramatically reduces it by several orders of magnitude is just not the way to go.

If somebody solves this problem it'll mean they solved much more interesting problem – because it'll mean you can locally uptrain model and inject this knowledge into bigger one arbitrarily.

But given that they don't seem to have worked on it since, I guess it wasn't too successful. But maybe there is a way

There is nothing stopping you from distributing assembly/machine code for CPU instructions, yet nobody does it because it doesn't make sense from performance perspective.

Or amazon driving truck from one depo to other to unload one package at a time to "distribute" unloading because "distributing = faster".

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."

It's only a question of scale and not of any principles.