Develop on a tiny slice of you dataset locally of you must. Then train properly elsewhere (in the cloud of your choice)

You could do this via Jupiter notebooks and visual studio code.

I develop my code directly on the HPC cluster. What do you even mean with testing your code instantly? You don't have to run a batch job every time you need to test something. You can use interactive mode.

You will probably keep your dataset on Cloud storage and Google Cloud has much higher Network speed, it makes things much faster than working locally

Obviously if you have your own local data center then you would use that

I can always test out my code instantly. With the vs code remote ssh plugin it's exactly like running everything locally, but in reality only the GUI runs on my laptop. The necessary connection speeds are tiny compared to what you'd need on the go to sync models and datasets.