Also depends on the complexity of the algorithm (specifically thinking of large neural networks). We have a model that requires 8 A100s for training due to the size of the activations. No way to replicate that on a local machine and have it train successfully in any reasonable time frame.

However the complexity of the algorithm many times scales with the size of the dataset, either the full corpus or the size of individual examples.