Would this work? I can't see meaningful work being down on the dataset piecewise.

Don't you need the load the entire parameter set to backpropagate and potentially at quite high precision. Then probably load less than 8gb as typical max vram. And then finally send that all back.

I think you really need dedicated data centres if only to move the parameters around and load the entire thing at once.